Lieu Zheng Hong

My reading journey

2026-08-30T00:00:00Z

My experience with aristocratic tutoring

2026-08-30T00:00:00Z

One simple trick to have good dates

2026-08-30T00:00:00Z

I think men (maybe both men and women, but I can only speak for men) stress too much about how to go on a date.

I know a friend who has a "playbook" for dates: he's optimised the exact restaurant to bring the girl (same restaurant every time), walk the same route after back home etc. Yeah, if you do that, you're going to hate dating, obviously.

Here's one simple trick that helped me a lot:

Instead of thinking of a date as an audition, where you think "what would she like to do", think, "what would I like to do", and invite her along!

Go to a restaurant that YOU want to go. Do an activity that YOU want to do. And then just invite her along!

OK, some caveats are in order here: to be clear, it's probably not a good idea to bring her to your weekly FNM (the smell can be quite something). But you contain multitudes, right? Go do something that you've always wanted to try but never did.

Scattered thoughts and changes on turning 30

2026-08-03T00:00:00Z

What has changed

I'm more zen and more chill

Perspective

And I don't think this is something that can be speedrun or something that can change just reading advice. I've always been a bit of an anxious person

Less obsessed about "productivity "

I kind of automatically stopped playing computer games

I'm kinder to myself now

What I've learned

Not blundering and not giving up is 90% of success

In my field of highly competitive startups, i see a lot of people just give up or implode, ...

The corollary is not to blunder. As I write in Advice for finalists, the body affects the mind, and:

when your judgement is affected, the first thing to go is the ability to judge that your judgement is affected. Learn to listen to your body, because it really, really affects the mind.

and yeah, I'd echo that

Sometimes people look way more impressive than they actually are

People project success

People's external state have little bearing with their internal conviction and I find it is difficult to ascertain who "got that dawg in them". The people who are sounding extremely outwardly confident may not be the ones with the intestinal fortitude to push through and succeed down the line.

I remember

We really underestimate the power of consistent compounding

Fitness, relationships, language learning, skill acquisition, all of these benefit

put in the work every day

Women will naturally like you

this one is a big change

You should not always "maximise optionality"

The advice to young people is often "maximise optionality"; I actually think this is retarded advice, sorry; or rather, it's not bad advice per se, but it's misused to gaslight young people to take up soul-sucking jobs at McKinsey or Bain and co.

In 2022 I came off a post-Covid bender, and fresh off raising money for my startup, started living the life of a "digital nomad". In the space of five months I was in Darmstadt, Geneva, Glasgow, Reykjavik, Brussels, Croatia, London, then Oxford.

There are certain things you can only do if you commit and compound. I'll give an example. If you look at all the startup literature out there, the YC stuff, they all primarily focus on the early stage finding PMF part. And the kind of implicit assumption is, by elision, that once you find PMF, the rest is like easy. And to be clear, finding PMF is the hardest and most critical part, without a product people will pay for, you have nothing. But I've realised now, with the benefits of already having found PMF, that there are yet more challenges to overcome. The challenges are different and more interesting and I enjoy my job now more (five years in) than when I started. If you do a startup for two years and exit and restart, you never get the chance to face and tackle these types of "scaling" or "post-PMF" problems. So that is an example of something that only happens when you commit and compound for a long time.

Think about what success looks like and whether or not you want it

Here's something that's happened quite a few times: a bright young founder asks me whether they should start a startup that does XYZ, where XYZ is the hot new thing (at time of writing, AI, prediction markets). And I tell them:

Things always take longer than they should; so before you start something, try to visualise what success looks like, and make sure you want it! If you succeed, you'll be stuck doing that thing for a long time, so make sure that that's something you actually want to be doing. The worst thing is to wake up and be trapped doing something you hate, even (especially) if it's going well.

You might be stuck in a good business doing bad and that will come at the price of your soul.

What has not changed

Wanting to build things that matter

Sometime in 2019 I started saying that I wanted to do "difficult, interesting, and impactful work". Reason being, if it's not interesting, I can't bring myself to do it. If it's not impactful, what's the point of doing it. And if it's not difficult, let someone else do it.

With age, I have changed my view: not that I don't think "difficult, impactful and interesting" is correct, but rather, I think the most important thing, bar none, is focus on "impactful".

Why is that? Because I realise that everything in this world is more difficult that it seems. The skill ceiling is very high. And if it's difficult and impactful, then it will naturally become very interesting to you. You will go into a flow state

What makes work impactful for me, is to work on something I know is positive-sum, building on solid foundations, unharried, sustainable growth over the months and years. See the above: think about what success looks like and whether or not you want it.

songs I like

2026-05-03T00:00:00Z

songs from CDs

things I heard on the radio

can sing OK

借
淒美地

good for practice (in this weird intermediate range)

大魚
奉勸
如果你也聽說
我走後－ en
歌謠
如果雨之後

Pixel Art on NFC cards

2026-05-03T00:00:00Z

I think NFC tags are really cool! I like how physical and embodied they are: a boon in these hyperdigital times. They're a lot more tactile and easy to use (you just tap them) compared to a QR code, where you have to open up your camera app, line it up with the QR code, wait a while for the QR code to be detected, then click the link.

NFC tags can store anywhere between 100 bytes to 8KB but the ones I bought have 888 bytes capacity. They cost around $2.50 per card from SGBotics but I'm sure you could get them cheaper. 888 bytes is not a lot of space: it's just enough for a long tweet or very short letter. We decided on pixel art! I built a very simple pixel art editor and viewer that lets you create your own tiny (20x32) pixel art and store it in the NFC card. This way, you have "art on a card".

Here's an example: ebb's "a clearing"

How does this work? Well, first of all, the image (and title/artist) are encoded into bytes, which can be represented as a string. The image above is equivalent to the following string, also called a fragment:

CgNhIGNsZWFyaW5nZWJic5znOc5znOc5znOc5znOe5z3Oc5znOc5znOe5znOc57nOc9znOc57nuc9z3Oc5znOc5zzuc9z3ie9z3Ocd7nOc573O25z3Pc5znOc97nPe5z3Pc57nPe5znO29znuc9z3Oc9znPe973Pe9z9ue9z3vcB73Pe573Pe573vA973uc573vc973ve5z3ve5z37ed73vA97nPA973ge973Pe9z3Oe5z3ve94Hve9z3vc573ve4DnPe97wPftz3ge973Pc97tve9z3ve973ve973ve573ve973vO973vA973ve953ve973ve973ve973ve9z3ve973vfx73ve97nPe973ve_R-Pe973Oc973ve9BzjOd73vc6rnve96UJ0fT3ve51aue9761SOp61e9z3VdV1avrWo61rWrnurWrqvrWta1HWtaudWr6ue5761rWr61rV1a1rV1XOfWrq1dVz61rWtauq51avrVznurWr61rWrnuc5zq1dV9a1fWta1c9zq1c5zq1rWtavq-ta1fV1a1rWtQ

We add this to my editor or viewer which can then read the fragment and display it in the browser.

The full URL looks like this: https://lieu.gg/nfc_pixel/viewer/#CgNhIGNsZWFyaW5nZWJic5znOc5znOc5znOc5znOe5z3Oc5znOc5znOe5znOc57nOc9znOc57nuc9z3Oc5znOc5zzuc9z3ie9z3Ocd7nOc573O25z3Pc5znOc97nPe5z3Pc57nPe5znO29znuc9z3Oc9znPe973Pe9z9ue9z3vcB73Pe573Pe573vA973uc573vc973ve5z3ve5z37ed73vA97nPA973ge973Pe9z3Oe5z3ve94Hve9z3vc573ve4DnPe97wPftz3ge973Pc97tve9z3ve973ve973ve573ve973vO973vA973ve953ve973ve973ve973ve9z3ve973vfx73ve97nPe973ve_R-Pe973Oc973ve9BzjOd73vc6rnve96UJ0fT3ve51aue9761SOp61e9z3VdV1avrWo61rWrnurWrqvrWta1HWtaudWr6ue5761rWr61rV1a1rV1XOfWrq1dVz61rWtauq51avrVznurWr61rWrnuc5zq1dV9a1fWta1c9zq1c5zq1rWtavq-ta1fV1a1rWtQ

But where does NFC come in?

The next step is to write the URL to the NFC card. Modern smartphones can read and write to NFC cards for free! We use an app called "NFC Tools" to do that. It takes about two seconds to write that URL (which is 579 bytes or so) to the NFC card.

If the data loaded on the NFC card is a URL, both Android and iOS devices will automatically open it in the browser by default. So when the user taps the NFC card on their phone, it takes them directly to the image! You now have art-on-a-card.

Last Saturday, Shashvat and I gave an interactive workshop called "Pixel Art Jam" to try out these concepts. We got them to download the app, make some art, write them onto their NFC cards, then share them with other attendees -- all in ten minutes. Here are some of the folks' works from TWC:

Clotho's "caw caw wizard"

Caleb's "sunny trees"

anonymous's "untitled"

ebb's "a clearing"

I am very, very impressed by how awesome these works are and how much people could do in just 5 minutes! These are awesome, truly.

Future work

I don't know if this could be done in 888 bytes, but I think it would be really cool to build the editor and viewer (or at least the viewer) in the URL itself, so the card could work completely offline.

Growing up in the 90s

2026-05-03T00:00:00Z

Sourced from /r/Singapore (permalink). made me feel something

Only 90s kid will remember, dating 90s ah lians is the best thing ever. Always look very chio in their rebond hair and heavy eyeliner. I like it when they move their head and the hair move around like plastic. Always wear FBT shorts, because she say she tomboy and 女强人, but both of you know its anything but that. Like to wear halter top and/or crop top, very skinny, chicken rice $2 only eat half the plate. Quite easy to stead, and very fast can do a lot liao. Initially also very exciting, move jin fast, she always call you 老公 and talk about marriage like 2 weeks in. Must buy ring from 77th street, wear around like engagement ring, jin romantic. Go clubbing together at sparks disco, see her 摇头 and her plastic rebond hair flying everywhere, jin wild, look like a primal goddess, and hear her shout the 369 chant to crazy baby jin fierce. Must have warning, because all her 乾哥 will be after her, and if she turn cold after about a month in, expect that within a few days, she will break up with you and stead her 乾哥. That was my experience with ah lians, all 3 start and end exactly the same, machiam photocopy, but I had a very good time, 10/10.

You a 90s kid sia. I program 眉飞色舞 and para para for her 3310 ringtone. Buy her phone charm some more to hang on the phone, one was a heart, other one forgot liao, I think got a slam dunk one. Their phone charm is larger than their phone and take like half their bag

Actually neoprint is one of the best things you can do in the 90s, once you close the curtain, can do anything in there. beyond all the 变态 things, actually do with friends is very fun, I still got a lot of the old neoprints with friends, some sausage fest, some with the girls I used to like, all the stupid hair and fashion, everything. If you look at it, you can remember the exact event you took the photo. Damn ex. somemore, I remember starting is like $6, and you want the very power one have to spend up to $12 also got. Remember always have to wait for someone to cut it out and divide among friends. Now you just selfie, forget to post, cos not glam enough, then lose the memory when you change the phone.

Dunno is it nostalgia lens or what, 90s seem to be the best era, everything also got, and not so digital, have to talk face to face. Now can laugh laugh at it, but remember last time you jio the ah lian, she say yes, also feel very very happy. Before you say the 你要做我的 stead 吗? feel jin nervous. Got some say no, then feel very sad, but ok, go pak some KOF in arcade, then feel good liao. Feel sian, spend a bit go eat zhi char at kopi tiam, then go kai kai, a lot of shops, very bright lights, as long as you got your friends everything also stready. Before go out, must make hair nice nice, put the sharp comb, if the group got the girl you like, then even more exciting, got chance talk to her, other ppl walk in front, you talk to her and slow down your pace until you all last, or even better, your buddies purposely lose both of you. Even go bookshop also fun, that time the books all more colourful and exciting, everything you want to learn must buy one, cannot find on the internet. Even when bored, still must go out kai kai. Still remember me and my best friend go kai kai at the old bayshore there, because too bored, the lights everything super romantic, but its just 2 guys walking, feel very romantic, but look at him feel very err xin, haha.

Now whole day worry mortgage, worry children, worry job, worry COVID, si bei sian. Everyday stress, everyday bored, not even in the same country as my 兄弟. Last time I went back all uncle liao, I talk about these things they laugh at me ask me I how old liao. After that even go back also never jio them, feel like everyone change, and feel like only I cherish these kind of childish things

video games are IQ shredders

2025-07-09T00:00:00Z

Tetris was invented in 1985, it came out on the NES in 1989, but the best way to play was only discovered in 2021.1 Previously, players would just try to tap the buttons really fast (“hypertapping”), until a 15-year-old named Christopher “CheeZ” Martinez realized that you could actually press the buttons faster if you roll your fingers across the back of the controller (“rolling”). CheeZ went on to set world records using his technique, but he wasn’t on top for long. Other players soon perfected their rolls, and CheeZ lost in a first-round upset at the 2022 Classic Tetris World Championship to another “roller”, a 48th-seed named BirbWizard.

I love this because it shows how low-hanging discoveries can just sit there for decades without anyone seeing them, even when thousands of dollars are on the line. (Seriously—the first place finisher in the 2024 championships won $10k.) People spent 40 years trying to tap buttons faster without ever realizing they should be tapping the other side of the controller instead.

But I also hate this, because:

What's the problem with this?

Why are Roger Federer and Christiano Ronaldo uber-cool, Magnus Carlsen somewhat cool, and "CheeZ" and "BirbWizard" not cool at all?

[09/07, 16:57] lieu: here’s a possible synthesis of my worldview, why I think it’s great to be good at “physical” sport but a big waste to be good at mind sport or esport, and why that Twitter screenshot resonates with me.

in our world, you can either have a good brain or you can have a good body (or both of course). Modern society (let’s leave the upcoming AI-apocalypse aside for now) gives great returns to having a good brain, near zero returns to having a good body (since we have mechanisation, industrialisation, what have you). And hence if you’re one of those with a good body, the only way you can take advantage of that talent in our modern world is to do those sport things: running fast in straight line, basketball, etc.

Whereas people who are good at chess and NES Tetris are obviously “good brainers”, they have so many more options to do something actually “useful” and “productive” with their talents. The ones that are being oneshot by video games or nerd games are bad because the counterfactual potentially is that they start some brainy startup or do a PhD or whatever, and instead they do this stuff

[09/07, 17:03] lieu: A type I IQ shredder was defined by Spandrell:

How many bright Indians and bright Chinese are there, Harry? Surely they are not infinite. And what will they do in Singapore? Well, engage in the finance and marketing rat-race and depress their fertility to 0.78, wasting valuable genes just so your property prices don't go down. Singapore is an IQ shredder.

Type I IQ shredders shred intelligence genes. They are very dangerous for the long term because they decrease the overall intelligence of the next generation. Type II IQ shredders destroy intellectual output now by distracting high IQ people from real pursuits, or via other means.

Tlon is a good example of a type II IQ shredder. If Tlon worship doesn’t effect high IQ reproduction, then it is only a type II IQ shredder, because it grinds actual intellectual output to a halt by distracting everyone intelligent. Likewise, if everyone went to San Francisco or Singapore to improve human knowledge, but they didn’t reproduce due to housing prices, density, liberal culture, etc, then San Francisco and Singapore would only be type I IQ shredders, because they adequately use the intelligence they are given, but deprive it from the next generation.

In reality, Singapore, San Francisco, et al. are both type I and type II IQ shredders, because not only do the people who go there suffer from reduced fertility — they also spend all day interacting with the intellectual Matrix, the Tlon that are computers.

[09/07, 17:07] lieu: > Type II IQ shredders destroy intellectual output now by distracting high IQ people from real pursuits, or via other means. Tlon is a good example of a type II IQ shredder. If Tlon worship doesn’t effect high IQ reproduction, then it is only a type II IQ shredder, because it grinds actual intellectual output to a halt by distracting everyone intelligent. Likewise, if everyone went to San Francisco or Singapore to improve human knowledge, but they didn’t reproduce due to housing prices, density, liberal culture, etc, then San Francisco and Singapore would only be type I IQ shredders, because they adequately use the intelligence they are given, but deprive it from the next generation.

[09/07, 17:07] lieu: the point is that video games are a type 2 IQ shredder

[09/07, 17:08] lieu: if you devote your time to speedrunning super mario bros 2 or whatever

[09/07, 17:08] lieu: that’s a huge shredder

[09/07, 17:08] lieu: actually i think this concept is nontrivial, i should write about it

[09/07, 17:08] lieu: like “why is sports cool, why is vidya lame"

[09/07, 17:08] lieu: not just lame but why is it a waste

[09/07, 17:08] lieu: because of the counterfactual

improving the site (2025)

2025-07-06T00:00:00Z

previous posts in this entry:

This year, I made two large architectural changes, things I've wanted to do for a long time.

content-presentation separation

The largest architectural change is a clear separation between content and presentation. When I first started this website it was a Jekyll static site, then Eleventy. Down the line I wanted to make it a Obsidian vault as well, and I did, but due to all of this historical baggage it was never quite done right, many vestigial files, etc.

I refactored the Github repo to have a very clear structure: on the root level we have vault and views, where vault is the Obsidian vault (content layer), while views is the presentation layer.

lieu/
├── vault/                           # 📁 CONTENT LAYER
└── views/                           # 🎨 PRESENTATION LAYER
	└── eleventy/                    # Eleventy SSG implementation

Conceptually, I want the vault to be "pure": it has all my notes and thoughts, and doesn't concern itself with presentation. The website is one of many potential views over my vault. Completely separating content and presentation lets me change or add new views if I see fit -- if e.g. Eleventy stops being supported, or I want to build a server-side rendered second brain with full text search over my private posts -- without having to change anything about the vault.

The other advantage is it also lets me back up my Obsidian vault with Obsidian Sync, which means that I can read and draft posts on my phone with the Obsidian app.

persistent, global blob storage

I want my second brain to store not just my written thoughts, but also my memories in the form of PDFs, images, videos, etc. I started putting all these files in a folder called /img inside the root folder, and backing it all up on Git. Of course, this is a bad idea, and very soon I had to explore alternative solutions.

For the past few years I used Git LFS, and maybe it's a skill issue on my part, but it's entirely horrible. I've tried many times to make it work, it just sucks, it's so painful, I eventually decided to rip it out entirely (losing all my old commit history).

So I decided to build a global blob storage service that

stores all my data (videos, photos, PDFs, ...)
serves it up at a persistent global URL (e.g. files.lieu.gg/img/group_testing/queue.png)
allows me to upload data (from blob-service.lieu.gg)
automatically captures new data from my phone and desktop (e.g phone screenshots or screen recordings, desktop screenshots, pasted images..)

The first three points are done with Cloudflare R2 and a Cloudflare worker, pretty much all done by Claude. That all works very well. The last one is a work in progress, and requires a bunch of different scripts: for example, Obsidian stores all pasted images in the local folder, so I had to write a hook that intercepts that paste, move the file to a temporary folder, upload it to the R2 bucket, delete the file, etc. For the phone I use Tasker.

downstream cleanups

Lots of smaller cleanups needed, some of which were downstream of the infra change. The collections system / system of filters I had used to categorise posts on the website was simplified and rewritten. I needed to remove layout:base from the front matter of all my Markdown files (because layout:base is a purely presentation consideration).

I migrated all /img/... paths to https://files.lieu.gg/... paths. This would have been painful before but Claude Code did a really good job at it, it wrote a bunch of not-great regexes, but if you check its work and tell it it's missed a bunch of files, it can eventually do it

claude code is incredible

This is the first improvement on my site that was AI-assisted! I guess this must mark a sea change of some sort.

Claude Code is incredible, really.

Nothing that hasn't been said already, but two observations:

It's surprisingly good at brainstorming solutions, if you're smart and willing to push back. It will suggest a bunch of frameworks -- I would strongly recommend not doing that (as of June 2025--who knows what will happen in just a few months!). These new trendy frameworks are not lindy enough, they don't have a sufficiently large corpus for AI to learn from, the code Claude will write is going to be all wrong.
Claude Code is good for work, but incredible for personal projects. Its biggest strength is getting things done that you wouldn't have done otherwise. I have so many "todos" and "improvements" to make but I know it's going to take a few days or even weeks, so I push them back indefinitely. For example I have wanted to build a global blob store for AGES, but the pain of reading through the documentation, implementing it, migrating all the paths, etc., I could definitely have done it myself, but it would have needed a couple of weekends, and so I always pushed it back. Because Claude can write decent-to-good code so quick, it can get it done in hours instead of days. This reduces the activation energy and offers what is in effect an "infinity" productivity improvement, because the counterfactual is that these improvements just don't get built without it.
It's incredible for writing documentation too! I got it to write the CHANGELOG after making some changes, check out a sample entry:

[2025-06-21] - Obsidian Link Compatibility

Added

Automatic Link Conversion: Created markdown-it-obsidian-links.js plugin to automatically convert Obsidian-style relative markdown links to Eleventy-compatible absolute paths during build

Plugin Integration: Integrated the link conversion plugin into Eleventy's markdown processing pipeline

Fixed

Obsidian Link Compatibility: Resolved incompatibility between Obsidian's relative link format ([text](filename.md)) and Eleventy's expected absolute path format ([text](/filename))

Content Consistency: Converted all manually converted absolute path links back to Obsidian-style relative paths for consistent authoring experience

Changed

Link Format Standardization: All content now consistently uses Obsidian's native relative link format (filename.md) while the plugin handles automatic conversion to absolute paths (/filename) during build

Date-based Link Conversion: Converted date-based permalink URLs (e.g., /2020/08/03/blog-rebuild-2020/) back to markdown file references (2020-08-03-blog-rebuild-2020.md) to leverage Eleventy's permalink system

Technical Details: The plugin operates during markdown processing, detecting relative .md links and converting them to absolute paths while preserving external URLs and non-markdown links. This solution eliminates the need for manual link format maintenance across 78+ affected files while ensuring seamless integration with Eleventy's routing system.

I think this is pretty awesome, it can make the changes AND document it. It is a bit verbose, but it's a lot better than what I would have written myself (mainly a patience thing)

further work

(bugfix) current rsync cronjob sucks and breaks often
(bugfix) tasker doesn't work well with photos taken with cameras
full-text search of the entire website
canvas display support
comment feature, ideally with threads

conclusion

I now have the following:

data store

blob storage, running at blob-service.lieu.gg and serving files at files.lieu.gg and
second brain (Obsidian vault + GitHub repo)
secrets vault (TODO)

views

lieuzhenghong.com or lieu.gg (static site, public by default)
(in future) private view of my second brain, with full-text search over all my private notes
(maybe, in future) offer services like semi-custodial crypto wallet for friends and family

Appendix

setting up blob storage

R2 and cron

Set up a Cloudflare account, set up R2 Object Storage

Set up rclone on the desktop

Set up bisync:

rclone bisync ~/Documents/lieu-blobs/ r2:lieu-files/ --resync for first time, then

Set up cronjob to sync

crontab -e */2 * * * * /opt/homebrew/bin/rclone bisync ~/Documents/lieu-blobs/ r2:lieu-files/

log show --predicate 'process == "cron"' --info --last 1d

two types of ideas

2025-06-21T00:00:00Z

there are two kinds of ideas: those that should be scrutinised, and those that should be coddled.

some ideas present clear costs against relatively predictable benefits. they benefit from being picked apart – from a transparent and strict evaluation of costs and benefits, and a thorough accounting of failure modes and unexpected consequences.

on the other hand, the best product ideas are delicate, fragile seedlings germinated from one highly counterintuitive insight. they almost always sound stupid and half-baked and they lack critical detail. The ROI is unclear ("this could be huge, but..."). these ideas should be coddled.

my own business was built upon one of those counterintuitive insights, and I'm not sure that insight would have survived harsh scrutiny.

most people lock themselves into one mode, either bearish scepticism or unquestioning support. it's probably a personality trait as well – some people are amazing at enumerating all the things that could go wrong, while others can hone in on that one magical kernel and pull out something great from an otherwise shitty idea. the best people I know have a keen sense of what type of idea something is, and modulate appropriately what type of feedback to give

scrutiny is harsh, direct sunlight – great for a tall tree, but too much for a fragile seedling. Some decisions need the midday sun; others need a nursery.

PG, How to Get Startup Ideas

A game 10 years in the making

2025-06-19T00:00:00Z

A game 10 years in the making, started from Hunter Killer (CKY's game) in 2016, then my version in 2022. Then I themed it differently and called it "Bulge", and built it for the 2022 GMTK game jam with dice instead of cards. Now it's a dice placement game instead.

Early prototype here

Inspired by Sagrada aesthetically. Also, LLMs have really grown by leaps and bounds, incredible

Prevailing Winds

A strategic dice placement game

In the toxic skies above a poisoned world, airship captains have developed a peculiar tradition. When negotiations fail and direct conflict seems inevitable, they settle disputes through an ancient game of dice and grid. The game itself is simple - place dice, score patterns - but how one plays reveals everything about their faction's soul.

The tradition emerged from necessity. In the early days after the Collapse, when every faction was desperate and trigger-happy, actual combat threatened to destroy what little of humanity remained. The game became a way to resolve disputes without bloodshed - though the outcomes often determined trade routes, salvage rights, or even the fate of captured ships. What began as pragmatic conflict resolution evolved into something deeper: a ritual that reveals the philosophical core of each faction.

Every captain knows that their opponents will judge not just their final score, but how they achieved it. Every placement echoes a fundamental answer to survival: Do you scatter and adapt like the Outcasts, refusing all authority even as stronger leaders emerge? Do you build unified foundations like the Settlers, perfecting humanity through controlled breeding? Do you heal the world like the Growers, engineering nature back to life? Do you impose order like the Military, wielding pre-Collapse might in a lawless sky? Do you gather knowledge like the Seekers, preparing humanity's escape to the stars? Or do you accumulate power like the Merchants, treating the apocalypse as a business opportunity?

Victory in Prevailing Winds isn't just about points - it's about proving your way of life superior. Each captain carries their faction's honor into every placement, every calculated risk. The dice don't lie, and neither does the philosophy they reveal.

Equipment

36 dice (6 colors, 6 dice per color)
Dice bag (opaque)
Scoring card deck
Captain cards
3 Zone coasters (numbered I, II, III)
6x6 grid board

Setup

Choose Captains: Each player selects a captain card, revealing their signature scoring pattern and ability.
Draft Scoring Cards:
- Deal 5 scoring cards to each player
- Each player chooses 4 cards to keep, discards 1
- Each player reveals 2 cards of their choice, keeps 2 hidden
- Players now have: 1 signature pattern (captain) + 2 revealed cards + 2 hidden cards
Prepare Zones:
- Put all dice in the bag and shake
- Draw one die, roll it, place on Zone I coaster
- Repeat for Zones II and III

Gameplay

Players alternate turns. On your turn:

Choose a Zone: Starting with Zone I, decide whether to take the dice from this zone
- Zone I & II: Optional - you may choose to skip and move to the next zone
  - If you skip: Draw one die from the bag, roll it, and add it to the skipped zone
- Zone III: Mandatory - you must take these dice
Deploy Dice: If you take dice from a zone, place all of them on empty squares of the shared 6x6 atmospheric grid
- All dice become neutral once deployed - no captain ownership
Refill Zone: Draw one die from the bag, roll it, and place it on the zone you just emptied
Captain Ability: You may use your captain's once-per-engagement ability at the appropriate timing

End Game & Scoring

The game ends when the grid is completely filled (36 dice placed).

Each captain scores points from:

Their signature pattern
Their 4 drafted scoring cards (both revealed and hidden)

The captain with the highest total score claims victory.

Key Mechanics

Shared Grid: Both captains place dice on the same grid, but dice have no owner once placed
Asymmetric Scoring: Captains score the same final grid using completely different patterns
Zone Risk/Reward: Earlier zones are optional but later zones are mandatory
Information Warfare: Hidden scoring cards create uncertainty about opponent's objectives
Philosophical Expression: Each captain's approach to play reveals their faction's worldview

Faction Captains

Captain "Red" Vasquez (The Outcasts)

Signature Pattern: "Exile's Domain": 2 points for each red die on an edge, 4 points for each red die in a corner
Ability: Once per game, instead of taking dice from a Zone, you may take up to 1 die from each Zone. Then replenish each Zone you took from. Then take your turn as normal.
Style: "Live free or die."

Comrade Chen (The Settlers)

Signature Pattern: "Genetic Purity": 10 points for each 2x2 square of white dice.
Ability: Once per game, instead of taking dice from a Zone, draw 3 dice from the bag. Choose any one die, set it to your desired face, then place it on the board. Return the dice you didn't choose back to the bag.
Style: "Unity is strength."

Engineer Mika "Bolts" Tanaka (The Growers)

Signature Pattern: "Natural Progression": For each sequence of consecutive green dice starting from 1, score points equal to the sum of those numbers. Multiple sequences can exist simultaneously.
- (Example: One 1-2-3 sequence scores 6 points, plus one 1-2 sequence scores 3 points, for 9 total)
Ability: Once per game, before taking your turn, you may swap any number of dice from any of the Zones. The number of dice in each Zone must be the same after your swaps. Then take your turn as normal.
Style: "Natura non facit saltus."

Captain Susan Bates (The Military)

Signature Pattern: "Maximum Firepower": 3 points for each black die that shows 6
Ability: Once per game, before taking your turn, you may reroll all dice across all Zones that show 1s or 2s. Then take your turn as normal.
Style: "Overwhelming force, decisively concentrated."

Prof. Gennady Mitrichev (The Seekers)

Signature Pattern: "Intelligence Hub": 3 points for each blue die where all adjacent dice show different colors and different numbers
Ability: Once per game, before taking your turn, draw three dice from the dice bag. Roll them and place one in each zone (your choice). Then take your turn as normal.
Style: "Knowledge is power."

Outer Prince Wangxian (The Merchants)

Signature Pattern: "Profitable Arbitrage": 3 points for each purple die that's adjacent to both a 1 and a 6.
Ability: Once per game, after any player's turn, you may swap any die on the board with any die from any Zone.
Style: "浑水摸鱼"

Scoring Cards

High Roller: 1 point for each die showing 6
Edge Control: 1 point for each die on the border that has the same number as an adjacent border die
Twins: 2 points for each pair of identical adjacent dice (e.g. both red 1s)
($COLOUR) Corners: 1 point for each corner occupied by $COLOUR.
Spread Out: 3 points for each color that appears in all four quadrants of the board
Four Corners: 8 points if all corners are 4s.
Straight: 10 points for each row, column, or diagonal that shows 1-2-3-4-5-6
Rainbow Road: 10 points for each row, column, or diagonal that shows R-O-Y-G-B-P
Colour Purity: 12 points for each 3×2 rectangle that contains only one color
Red Tide: 15 points if any border row or column is all red.
Perfect Twos: 15 points if the center 2x2 square contains all 2s
Six Sixes: 25 points if exactly 6 dice show the number 6
Perfect Balance: 30 points if each number appears exactly 6 times

Faction Histories & Motivations

The Six Powers of the Toxic Skies

In the aftermath of the Collapse, when the surface world became a poisoned wasteland and humanity fled to the skies, six distinct philosophies emerged. Each faction represents not just a survival strategy, but a fundamental answer to the question: "What does it mean to be human when the world has ended?"

The Outcasts (Red)

"Freedom or death - we bow to no one."

History: The Outcasts aren't a true faction so much as everyone who couldn't or wouldn't fit into the others. Exiles, rebels, free spirits, and those who simply wanted to chart their own course. They rejected the safety of large colony ships for the freedom of small, fast vessels and the open sky.

Motivation: Fierce individualism and anarchist ideals. They believe that centralized authority was what caused the Collapse in the first place, and that true freedom comes from rejecting all forms of institutional control. Every captain is sovereign over their own ship.

Fleet Philosophy: Small, fast, self-sufficient vessels. Hit-and-run tactics. Survival through adaptability and improvisation.

Fundamental Contradictions: While they embrace libertarian anarchy and radical individualism, the harsh realities of survival inevitably lead to stronger leaders consolidating power. A growing coalition under a charismatic strongman threatens their core values - but their anarchist principles make it nearly impossible to organize effective resistance.

The Settlers (White)

"Strong bodies, pure purpose, solid foundations."

History: Descendants of survivalists and physical fitness enthusiasts who believed human enhancement was the key to reclaiming the world. They've spent generations perfecting the human form through selective breeding, genetic engineering, and intensive physical training.

Motivation: To resettle the poisoned surface world using genetically enhanced humans in environmental suits. They believe that with enough physical and mental conditioning, humanity can adapt to survive anywhere - even a toxic wasteland.

Fleet Philosophy: Large sailing ships with white and red colors. Defensive tactics focused on endurance and outlasting enemies rather than quick victories.

Fundamental Contradictions: They preach the sanctity of the human body while practicing eugenics and forced sterilization. They claim to value mental development and diverse thinking while being deeply intolerant of dissenting viewpoints. Their pursuit of human perfection has made them profoundly inhuman.

The Growers (Green)

"Heal the world, don't abandon it."

History: Originally part of the Knowledge faction, they split over a fundamental disagreement about humanity's future. While the Blues wanted to escape to the stars, the Greens believed in healing and terraforming the damaged world.

Motivation: Develop biological solutions to purify the toxic atmosphere and rehabilitate the surface. They're breeding extremophile organisms and engineering algae to metabolize the poisonous gases, working toward a day when the surface blooms again.

Fleet Philosophy: Bio-integrated ships that blur the line between vessel and organism. Patient, defensive strategies focused on long-term growth over short-term gains.

Fundamental Contradictions: They preach organic, natural growth and spiritual harmony while simultaneously engineering organisms at the genetic level and manipulating nature through advanced biotechnology. Their emphasis on individual spiritual enlightenment (often aided by psychedelics) conflicts with the massive coordinated effort their terraforming project requires.

The Military (Black)

"Order from chaos. Authority through strength."

History: The remnants of pre-Collapse military forces who maintained their command structure and discipline even as the world ended. They possess the most powerful warships - massive pre-Collapse vessels that they struggle to maintain.

Motivation: Restore law and order in the post-apocalyptic world by establishing their monopoly on legitimate force. They see themselves as the inheritors of proper government and the only hope for returning civilization to the skies.

Fleet Philosophy: Overwhelming firepower and shock tactics. Use superior technology and concentrated force to end conflicts quickly and decisively.

Fundamental Contradictions: Their irreplaceable ships are slowly failing and they can't agree whether to abandon their technological advantage or desperately search for ways to maintain it. They claim to protect civilization, but the governments they served no longer exist - they may just be preserving the forms of authority without any legitimate purpose.

The Seekers (Blue)

"Through hardships to the stars."

History: Born from universities, research institutions, and tech companies that survived the Collapse. They split from the Greens over whether to fix the world or transcend it, choosing the path of technological advancement and eventual escape.

Motivation: Salvage lost knowledge, develop advanced technology, and escape to another world. They believe Earth is beyond saving and that humanity's future lies among the stars.

Fleet Philosophy: High-tech vessels with advanced sensors and AI. Information warfare - know your enemy completely before engaging.

Fundamental Contradictions: They pride themselves on egalitarianism yet their austere lifestyle only attracts intellectual misfits, accidentally creating an elite caste. They're dedicated to preserving human knowledge while planning to abandon the world that created it. They claim rational problem-solving as their strength, yet their core strategy is escapism rather than solutions. Their utilitarian efficiency may suppress the very creativity and human diversity that drives innovation.

The Merchants (Purple)

"Blood, gold, and power - the eternal currencies."

History: Built around the flying harem palace of a pre-Collapse trillionaire. Now his descendants rule a hierarchical monarchy where everyone claims blood relation to the founder, creating brutal dynastic power struggles.

Motivation: Maintain their decadent empire through trade and exploitation. They view all non-royals as barbarians to be used for labor or breeding. The most beautiful women around are "collected" for the inner palace and never seen again. Outer princes eke out a living, usually as merchants; inner princes live lives of decadent luxury but fight brutal dynastic battles behind the scenes.

Fleet Philosophy: Opulent vessels that display wealth and power. Slow, grinding economic warfare positions for long-term advantage rather than direct confrontation.

Fundamental Contradictions: Their rhetoric of family obligation and noble blood conflicts with the brutal internal power struggles and dynastic backstabbing. They practice slavery, forced marriage, and sexual exploitation while claiming to honor familial bonds. Their isolationist xenophobia conflicts with their need for trade and fresh victims.

Each faction's approach to the dice grid reveals their deepest beliefs about survival, society, and the future of humanity. In Prevailing Winds, you don't just play a strategy - you embody a philosophy.

On growth

2025-06-14T00:00:00Z

if addiction is the progressive narrowing of the things that bring us joy, growth is the opposite: it's the progressive widening of the set of situations we can be comfortable in.

We are all prisoners to our own neuroses. The way to grow past that is to deliberately expose ourselves to uncomfortable situations.

My explicit goal in 2025 was to actively and aggressively seek out uncomfortable situations; to "surf the edge of my discomfort", and by doing so become more comfortable with discomfort. I wanted to deliberately seek out failure and put myself in positions to get rejected more. Some personal examples:

Forcing myself to speak only German in Berlin
Asking girls out in person
Walking around Singapore barefoot
Going to social dances (this was by far the hardest one! When I first started I really hated dancing; I have two left feet, I'm super uncoordinated, I was pretty awkward touching people)

Discomfort should be carefully calibrated -- too much and you'll be paralysed into inaction, too little and it won't grow you maximally. How I "titrated" my discomfort:

Learning German: I first started sounding out the letters to Celine, who (as a kind-hearted person and my partner at the time) was obligated to be patient with me. Then I signed myself up for an intensive German course in Singapore. Finally I signed up for an intensive German course in Berlin and forced myself to speak German to strangers.
Asking girls out in person: I started talking to random guys at the gym, just to build up that "talking to strangers muscle"; it's a pretty low-stakes environment. Then I started going up to people at parties and introducing myself. Finally I started asking girls out in appropriate contexts (maybe don't do it on the MRT, but after a class or an activity/meetup seemed to work well enough for me)
Walking around Singapore barefoot: I first started going around my neighbourhood shirtless, then started wearing a sarong out, then finally started going about my day barefoot.
Learning to dance: What helped was going with a bunch of my classmates. Even today I have to psych myself up, I have to muster up the activation energy to leave my bed, and then I'll do some self-talk: "I'll just ask one person to dance". And then I go there and I ask one person and then it becomes easier and easier!

Many times our neuroses turn out to be unjustified or exaggerated. People are a lot nicer than you think! Examples:

I thought that people would make fun of my german in berlin but it turned out everyone was mostly VERY patient and very helpful (sometimes they switched to english when they were tired babying me, which, I completely understand, but 99% of the time they were amazing)
I thought I was a weirdo wearing a sarong out and I was super self-conscious about it, but people just thought I was Burmese or something. It was fine! One time someone shouted to get my attention in the street, I turned around, it was this Indian man wearing a sarong as well, he was like "Good job!" gave me a thumbs up lmao.
- The barefoot stuff was a lot weirder, people would come up to me saying "where are your shoes", I just said "I left them at home".
Some girls turned me down, some girls said yes (yay!) but even when they turn me down they do it in a very friendly and kind way. And of course, even one girl saying yes was a HUGE win, and made my week.
I was MORTIFIED going into social dancing, but people were mostly very friendly/patient; the not patient ones were cordial enough, and the best ones came back to ask for dances on their own accord, even gave me feedback and taught me new moves

Things won't be perfect; not everyone is going to be OVERJOYED THAT YOU ARE TRYING TO LEARN GERMAN, for example. Some people will be a little "black face" or mean, but by and large it's going to be A LOT BETTER than you think.

Xenophon on Cyrus:

Of all Cyrus’s many qualities: willpower, strength, charisma, glibness, intelligence, handsomeness; Xenophon makes a point of emphasizing one in particular, and his choice might strike some readers as strange. It is this: “He did not run from being defeated into the refuge of not doing that in which he had been defeated.” Cyrus learned to love the feeling of failure, because failure means you’re facing a worthy challenge, failure means you haven’t set your sights too low, failure means you’ve encountered a stone hard enough to sharpen your own edge. He’s found a cognitive meta-tool, one of those secrets of the universe which, if you can actually internalize them, make you better at everything. Failure feels good to him rather than bad, is it any surprise he goes on to conquer the world? (source: REVIEW: The Education of Cyrus, by Xenophon)

Advice for finalists

2025-06-13T00:00:00Z

Advice often sucks! It encodes the advice-giver's biases, and it's never tailored for the individual. You may want to consider reversing any advice you read here. The golden mean applies.

Don't stress too much about finals

It doesn't matter as much as you think. Really. No one at my job even knows (and they certainly don't care) what university I went to or what modules I studied. Your work will speak for itself.

Your agency, your work ethic, and your ability to learn and grow from here on out will make much more of a difference going forward.

Beware the prestige premium

The more legible and "prestigious" something is, the more overrated it tends to be.

By legible, I mean anything with a highly formalised and competitive application process, anything with spring/summer internship, anything with a fresh grad intake, etc. The gobs of money you can make as a barrister, consultant, banker are highly legible, but I promise there are gobs of money to be made all over the world in domains you wouldn't and can't imagine.

If you can find a field that none of your peers are going into, you'll stand out a lot. Even better if you can find a highly unpopular field people actively denigrate. These fields also select for iconoclasts, whom I always find more interesting.

Your option set is a lot larger than you think

Your option set is circumscribed by your beliefs and by the people around you. In the army I somehow thought that the best thing I should do is apply for an Air Force scholarship. And in Oxford I considered doing a Masters' because everyone else seemed to be doing it.

We put up a lot of self-imposed barriers on what we think our options are. Oxford is a very academic place; it's hard not to consider academia when all your friends, professors etc. are doing it. And if you see your forebears doing finance, law, consulting, engineering, medicine, it's easy to follow in their footsteps, too. Don't forget the world is big, you have only seen a very small part of it.

Good people are always in demand

If you are good, there will always be a place for you. From what I've seen, good people never need to look for jobs; the moment they quit, there are a string of offers for them, they just pick their favourite one. (Of course, it helps to put yourself out there: see "Just ask for things".)

I joined a startup incubator program last year. Most of the companies failed, but pretty much everyone got snapped up by other companies, because they were good.

How to be good? Work on stuff you genuinely think is important and interesting, not what society tells you is lucrative or prestigious. This will let you surpass everyone effortlessly; a genuine interest in something gives you the superpower of being able to work all the time, without it feeling like work.

Don't conflate looking good with being good. Some founders post on Twitter about how hard they work, they'll focus on vanity metrics so they can raise bigger and bigger funding rounds, they'll go to lots of events and podcasts, they'll be Forbes 30 under 30... but none of this matters because they can't actually build. Resist the urge to play status games; just focus on being good instead.

Learn to build

The vast majority of legible, prestigious jobs seem not to be about building at all, but about managing, allocating, optimising.

Learn to build, not allocate capital.

Building is a lot more fun and a lot more fulfilling. Building is imposing your will on the world; it is drawing order out of chaos. Compared to a capital allocator, you have maximum agency. Would you rather watch porn or have sex?

If you learn to build, you can make things happen, and making things happen is a rare superpower. Having lots of money is great, but even people with money need good builders to make things happen.

Just ask for things

The rules of the game are a lot more malleable than you think. A lot of the time you can just ask for stuff and you might just get it!

There used to be a rule in PPE (maybe there still is) that if you wanted to do a Thesis in Politics, you had to do three other politics modules. But I didn't want to do three other politics modules. I thought it was set in stone (I mean, it's very explicitly written in the Students' Handbook), but I thought I'd punt anyway. So I emailed the Academic Registrar if I could do it. After a bit of back-and-forth (and with Bassel and Tia putting in a good word), it was approved???

Most jobs, internships etc., are not advertised. If you admire someone and their work, tell them so, and ask if you can work with them; I promise they'll be flattered. I've done this a few times already, and had many fruitful collaborations.

The trick to asking for things is not to get too attached to the outcome. Don't take "no" too personally -- it may not be about you. (And sometimes it is, but you're always growing, right?)

Treat the people whom you ask things from with the utmost respect and kindness. It's hard to say no, so make it as easy as you can for the other person. If you ask to work with someone, and they say no, maybe they don't have a vacancy, maybe you're not skilled enough yet, no worries, thank you for considering me in any case. If you ask someone out and they say no, okay that sucks, but hey, absolutely no worries at all, totally understand, respect your decision. And then tell yourself: it was a long shot anyway, I'm glad I asked!

Minimise regret

A lot of the time we don't regret doing things; we regret not having done things.

When I was in college there was a gorgeous girl called Rosanna. I had a big crush on her and we got along quite well, but I never asked her out, because the last guy she dated was a really tall, handsome, and posh guy who went to Eton, and I didn't think she'd deign to date a short Asian guy who went to Bukit Panjang Primary School. It turned out her last boyfriend was half-Asian and shorter than her, so I had ruled myself out for no reason. I still think about that sometimes. Don't let that be you!

Our fears, insecurities, and anxieties often paralyse us. Force yourself to bias towards action to counteract this natural tendency. When you do something, you may succeed (great). And if you fuck up, you will learn, and that's a great outcome too. When you choose inaction, there is no chance of succeeding, plus you have learned nothing. You miss 100% of the shots you don't take.

Lean into discomfort

Actively do stuff that makes you uncomfortable.

If addiction is the progressive narrowing of the set of things that give you pleasure, growth is the progressive widening of the set of situations you can be comfortable in.

The more you do stuff that makes you uncomfortable, the less uncomfortable those situations will become. Titrate slowly, take baby steps; you will make a lot of progress very quickly.

Body affects mind; mind affects body

I always thought of my brain as a Platonic-ideal thinking machine, with no relation to my body, always able to find some clever solution to intractable problems.

My startup was gearing up for a big public launch. I'd been working 7 day weeks for 3 weeks straight, with pretty much no sleep. I'd try to get to bed, but I'd find myself waking up in the middle of the night, heart racing, or have my mind spin in circles due to all the stuff we'd still need to do.

There was a lot of pushback on the launch due to some decisions we'd made, and after yet another all-nighter, I'd written a draft announcement reversing some of the more controversial decisions. I thought it was good work; an excellent way to thread the needle.

I sent the draft over to our launch partner for final comments. The founder of that partner company called me and said: under no circumstances should you send this out, you will completely destroy your reputation.

I am so glad I listened to him; he saved me from self-destructing my company.

My mentor once told me: when your judgement is affected, the first thing to go is the ability to judge that your judgement is affected. Learn to listen to your body, because it really, really affects the mind.

Take care of yourself!

Be kind to yourself and others

High-achiever types are often very harsh with themselves. I think it stems from some deep-seated insecurity of not being good enough.

I often find myself lying in bed reliving every single one of my mistakes; "I should have said this, I should have done that, if only..."

You are not perfect; you will fuck up, but you will improve. Be kind to, and patient with, yourself.

My self-talk is oftentimes very harsh. When I catch myself doing so, I say to myself: would I say this to a stranger or a friend? And the answer often is no.

Give yourself the space to celebrate your good work, don't keep rushing to the next thing.

If you admire someone for something, tell them about it! You will make their day.

Be humble

Growing up we're all taught to do the thing where we all glaze the guy who does well ("wah you damn upz sia") and the guy has to deflect everything ("no la no la lucky only"). This isn't actually humility; this is performative, this is wayang show.

Humility is not performative modesty. Humility is being rightfully proud of what you've achieved, and simultaneously recognising that you still have a long way to go and a lot to learn.

It really is about the friends you make along the way

There is no need for everyone to like you, you don't have to be friends with everyone. Polite, cordial, certainly; but some people are just not your tribe, and that's fine. Try your best to find your tribe: the people who share your same values, who match your freak. Then hold onto them and don't let go.

Some of your friends you'll admire because of their competence, others you will love because of their kind heart. You should try to find the thing you love in every one of your friends.

Be a good person

As you focus on being a good builder and executor, don't forget to be a good person, too. Are you loyal to your friends? Are you true to your word? Are you generous? Respectful? Kind? Compassionate? Are you doing any 亏心事?

As with anything else, being a good person is a learning process, too. I am selfish, judgmental, lazy, dishonest, impatient, rude, and a hundred other things, but I will always strive to be better. In the short time we are here on this earth, we should try to be like shooting stars -- lighting up the lives of those we pass by.

Braindump on language learning

2025-06-10T00:00:00Z

Deutsch Lernen

2023-09-10T00:00:00Z

[toc]

introduction

very rough notes, not meant for public consumption, it's where I dump everything

2023-09-10

die Gelegenheit die Gelegenheit sprechen zu können und uns mit längeren Texten befassen zu können ich fuhle mich verloren - i feel lost der? Arbeitsumfeld in den Medien das kommt darauf an - it depends Es kommt darauf an welchen Stil der musiker hat. Normaleweise lese ich keine nachtrichten, selbst auf Englisch. vorkommen Wenn man schnell zum Arbeit gehen muss, wurde man ehe ein Taxi nehmen anstatt zu Fuß zu gehen

2023-08-31

[x] abschnitt - a section
- [x] Als Rettungsschwimmer beobachte ich einen Abschnitt des Flusses.
[x] beweisen - to prove
- [x] In meinem Studium beweise ich viele Theorien.
[x] die Interessengebiet
- [x] Das wissenschaftliche Interessengebiet (das Fach, an dem man interessiert ist) von mir ist Wirtschaftswissenschaft, besonders die Spieltheorie.
[x] seit kurzem
- [x] Seit kurzem (in letzter Zeit) mache ich Calisthenics.
[x] ablegen -- to discard
[x] das Denkmuster -- the thought patterns
- [x] Wenn man Chinesisches Schach spielt, muss man die alten Denkmuster aus dem internationalen Schach ablegen.
[x] der bereich -- area
- [x] Im Chinesischen Schach kann der König nicht über seinen Bereich hinausgehen.
[x] hingehen, gehen
- [x] Die unterschied zwischen "hingehen" und "gehen" besteht darin, dass "hingehen" ein Ziel, einen Bestimmungsort braucht.
[x] erfordert -- to require
- [x] Lernen erfordert ständige Wiederholung.
[x] unvorsichtig -- careless
- [x] Ich bin sehr unvorsichtig. Ich hänge meine Figuren beim Schach immer auf.
[x] unüblich
- [x] Es ist unüblich, "kreuzen" anstatt "überqueren" zu sagen.
[x] überqueren: Wir überqueren die Straße.
[x] ertrinken -- to drown in
[x] Zeitlupe - in slow motion
- [x] Ich bin ein Rettungsschwimmer. Wenn eine Person ertrinkt, laufe ich wie in Zeitlupe um die Schwimmerin zu retten.
[x] ehreamtlich
- [x] Wenn man es ehrenamtlich (freiwillig) macht, erhält man kein Gehalt.
[x] die Nacktschnecke
- [x] Heute haben wir im Park eine Nacktschnecke gesehen. Sie war sehr lang und rot.
[x] der Schwert, Feinde, besiegen
- [x] Der Mann traniert mit einem Schwert, um seine Feinde zu besiegen.
[x] abnehmen: to lose weight
- [x] Nächstes Jahr will ich 5kg abnehmen.
[x] ausdauer, abgewogen
- [x] Es wird zwischen kraft und ausdauer abgewogen.
[x] zuständig für
- [x] Die Handlers sind zuständig für das Werfen der Frisbee.
[x] vorhersehbar
- [x] "Vertical Stack" ist relativ einfach zu spielen, aber vorhersehbar.
[x] höchstens
- [x] Ich kann höchstens 90 Tage in Europa bleiben.

faust

[x] wohl
[x] erscheinen, (ist erschienen) -- to publish
- [x] 1587 ist bei dem frankfurter buchdrucker das werk wohl zum ersten Mal erschienen.
[x] akten
- [x] In den Akten der Städte wird er als Nigromant genannt.
[ ] Sein Name soll Johann Faust gewesen sein
[x] der Teufel, holen (geholt),
- [x] Ob das eine Explosion war oder ob der Teufel ihn geholt hat, wissen wir natürlich nicht.
[x] Für Kunst scheint die Idee, seine Seele dem Teufel zu verschreiben, um Erfolg zu haben, besonders naheliegend zu sein.

wortschatz

[x] die Absage - the rejection
- [x] Je mehr Absagen ich bekam, desto frustrierter wurde ich naturlich.
[x] abwechlungsreich
- [x] Mein Job ist oft gefährlich, aber immer abwechslungsreich
[x] anstrengend: exhausting
- [ ]
[ ] berechtigt
[ ] die Berufserfahrung
[ ] die Bewerbung
[ ] sich durchkämpfen
[ ] entlassen
[ ] geeignet
[ ] genügend
[ ] die Herausforderung
[ ] klappen
[ ] der Lebenslauf
[ ] das Praktikum
[ ] die Stellenanzeige
[ ] übernehmen
[x] die Tätigkeit
[ ] der Verlag
[ ] vermitteln
[ ] der Vertrag
[ ] vielfältig
[ ] zahlreich
[x] angemessen
[ ] das Ansehen
[ ] anspruchsvoll
[ ] etw. bewältigen
[ ] die Finanzkrise
[ ] die Flexibilität
[ ] hauptsächlich
[ ] die Last
[ ] die Leistung
[ ] lukrativ
[ ] mangeln an
[ ] der Misserfolg
[ ] die Studie
[ ] der Verlust
[ ] verlässlich
[ ] vernünftig
[ ] der/die Vorgesetzte
[ ] ablenken von
[ ] die Geschäftsleitung
[ ] der Hammer
[ ] nahegelegen
[ ] die Säge
[ ] die Teambildung
[ ] das Teamevent
[ ] umrunden
[ ] voraussichtlich - wahrscheinlich, vermutlich
[ ] zwingen zu (jmd. zwingen etwas zu tun)
[ ] der Abschluss
[ ] die Anforderung
[ ] die Betreuung
[ ] einheitlich
[ ] der Eintrittstermin
[ ] die Kenntnis
[ ] einbringen in
[ ] die Fähigkeit
[ ] die Stellenausschreibung
[ ] teamfähig
[ ] übersichtlich
[ ] umfangreich
[ ] die Weiterbildung
[ ] zielorientiert
[ ] behandeln
[ ] Anforderungen erfüllen
[ ] jmd. fest anstellen
[ ] sich an die Arbeit machen
[ ] eine Aufgabe übernehmen
[ ] Druck ausüben
[ ] Fehler eingestehen
[ ] in Gefahr sein
[ ] sich im Job aufreiben
[ ] Konflikte lösen
[ ] Kontakte knüpfen
[ ] Sinn und Zweck von etw.
[ ] im Zusammenhang mit etw. stehen
[ ] zwei linke Hände haben
[ ] Es kommt darauf an
[ ] aktuelle sein
[ ] aufpassen
[ ] tippen
[ ] annehmen
[ ] zweiten Spalte
[ ] Zeugnis
[ ] nicht zwingend
[ ] zusätzlich
[ ] der Vekehr
[ ] der Verein
[ ] die Ernährung
[ ] der Konsum
[ ] das Krankenhaus
[ ] der Abschluss
[ ] das Studium
[ ] die Universität
[ ] die Wahlen
[ ] die Behörde
[ ] die Medien

2023-08-19

[x] überlegen / die überlegung
- [x] Ich überlege heute abend ins Kino zu gehen
[ ] nachweise
[x] ausgetreten
[x] auf jeden fall -- definitely? in any case
[x] Die Schere zwischen Arm und Reich n Singapu ist relativ groß.
[x] unter Drogen stehen

lieu, [19/08/2023 09:39]

Ischgl -- skiing spot in Austria.
Apres Ski
Vabali Spa -- in der Nahe von dem Hauptbahnhof

2023-08-16

[ ] vorteil
- [ ] Ich habe ein Vorteil davon, wenn ich Deutsch sprechen kann.
[ ] aussagekräftig - relevant
[ ] ausschließlich -- nur (exclusively)
[ ] Eigenschaften
[ ] Heutzutage schicken wir die Bewerbung per Mail mit Anhängen.
[ ] leistung v.s. leitung
[ ] stattfinden
- [ ] alle Veranstaltungen in englischer sprache stattfinden
[ ] worauf ist Marein Knoller in ihrem Anschreiben eingegangen.
[ ] Was Sie erzählen, sollte im Zusammenhang mit der Stelle stehen.
[ ] ich brauche die herausforderung

2023-08-15

[ ] übersichtlicher
[ ] Nicht nur das Jahr, sondern auch die Monate angeben, z. B. 06/13 oder Juni 13. Achten Sie darauf, dass die Datumsangaben einheitlich sind.
[ ] Mir Hammer und die Säge machen wir uns and die Arbeit, ohne von Mails oder Anrufen abgelenkt zu werden.
[ ] Sie hat angerufen ohne das Teamevent buchen zu bekommen.
[ ] Ich habe lange mit meinem Kollegen gesprochen, ohne dass ich ihn nicht überzeugen.

2023-08-13

[ ] sonst, diejenigen
- [ ] sonst stellen besonders diejenigen, die sich im Job aufreiben, irgendwann fest, dass das restliche Leben leidet.
[ ] etwas schaffen: to accomplish something
[x] schaffen: to do
[x] jedoch: however
- [x] Die Reise war großartig, jedoch auch anstrengend.
[x] sowohl, als auch
- [x] Für viele Stellen ist sowohl eine Ausbildung als auch genügend Berufserfahrung wichtig.
[ ] tätigkeit
[x] vorgesetzte: the boss
[x] der Umgang: das Firma
- [x] Vorgesetzte sollten ein angenehmes Arbeitsumfeld schaffen.
- [x] Vorgesetzte sollten weniger Fehler im Umgang mit ihren Mitarbeitern machen.
[x] Erwartungen erfüllen
- [x] Arbeitnehmer solltern sich deshalb öfter fragen, ob die Arbeit ihre Erwartungen erfüllt.
[ ] eine große Rolle spielen
[ ] eine reine Last sein
[ ] sich im Job aufreiben -- to wear oneself out on the job
[ ] etwas selbst in der Hand haben
[ ] das Vergnügen : the pleasure
[x] verlust von ansehen, mangelden
- [x] Wer arbeitslos ist, hat besonders mit dem Verlust von Ansehen und sozialen Kontakten und der mangelnden strukturierung des tages zu kämpfen.
[x] lockt
- [x] Dabei lockt auch das erwartete gute Gehalt.
- [x] Laut Ali lockt ein schnelles Auto die Frauen an.
[x] angemessen
- [x] Das Gehalt ist also gar nicht so entscheidend, sollte aber der Leistung angemessen sein.
[x] Stattdessen, unterstützt
- [x] Stattdessen sollten Mitarbeiter fachlich unterstützt und Konflikte schnell gelöst werden.
[x] beschäftigen sich, anspruchsvollen
- [x] Die meisten Menschen beschäftigen sich gern mit anspruchsvollen Aufgaben.
[x] zufrieden, vernünftigen
- [x] Besonders zufrieden sind Menschen, die ihren beruf aus vernünftigen gründen wie sicherheit wählen.

Ein schnelles Auto zieht Frauen an

2023-08-08

[x] die Betonung: stress, emphasis
- [x] Spanisch hat eine freie Betonung der Wörter.
[x] ständig -- immer, konstant, allzeit
- [x] Ella und David haben sich ständig gestritten.
[ ] erfüllen
- [x] Ich erfülle deinen Wunsch.
- [ ] Ich erfülle die Rolle eines Lehrers.
[x] ertragen: (aushalten)
- [x] Es gibt manchmal so viel Schönheit auf der Welt, dass ich sie fast nicht ertragen kann.
- [x] Wieviel kann ich noch ertragen?
[x] leiden
- [x] Deine Gesundheit wird darunter leiden.
- [x] Dafür muss ich heute leiden und leiden.
[x] geht (es) um
- [x] Wie gehen sie mit Kritik um?
- [x] In diesem Text geht es um Streit.
[x] Es reiz mich. Das gefällt mir.
[x] leihen: 借
- [x] Ich habe ihm Geld geliehen.
[x] mittlerweile: by this time
- [x] Sie versteht sich mittlerweile wieder gut mit ihrem Exfreund und manchmal hilft er ihr bei bürokratischen Problemen.
[x] entschlossen
- [x] Er war entschlossen, für Gerechtigkeit zu kämpfen.

2023-08-07

[x] träume von (Dat.)
- [x] Ben träumte schon lange von einem längeren Auslandaufenthalt.
[ ] bewarb sich bei (Dat.), bewarb sich um (Akk.)
- [x] Er bewarb sich bei einer italienischen Firma.
- [ ] Er bewarb sich um die Stelle.
[x] warten auf (Akk.)
- [x] Ich warte ungeduldig auf eine Antwort.
[x] eingeladen zu (Dat)
- [x] Ich habe eine Freundin zum Kaffee eingeladen.
[ ] verabreden mit (Dat)
- [ ] Am Wochenende verabreden ich mich oft mit meinen Kollegen.
[x] teilnehmen an (Dat)
- [x] Wenn ich mein Deutsch verbessern, kann ich besser an Besprechungen aktiv teilnehmen.
[x] sich gewohnen an (Akk.)
- [x] Ich habe mich schon gut an mein neues Leben in Berlin gewöhnt.
[x] verletzt:
- [x] Streit ist also ganz normal, solange er niemanden verletzt und eine Lösung bringt.
[x] beachten
- [x] Ich gebe Tipps, was man beachten kann, um der Konfliktlösung näher zu kommen.
[ ] Keine guten Aussichten auf Erfolg hat ein Streit, der mit einem Vorwurf beginnt.
- [ ] "Keine guten Aussichten auf Erfolg" - "No good prospect of success has an argument, ..."
[ ] Die dreckige Küche hat Ihren Streit ausgelöst?
- [ ] The dirty kitchen provoked your argument?
[ ] Erweitern Sie Ihre Kritik nicht auf andere Punkte, die sie eventuell auch noch stören wie Erziehungsfragen, Geld oder die Erledigung von Arbeiten.
- [ ] Extend not your criticism to other issues, that may also possibly be disturbing them, like parenting issues, money, or carrying out their work.
[x] Das erzeugt nur unnötige Spannung.
- [x] creates only unnecessary tension.
[x] Ich bin unter Druck steht.
- [x] Leute, die unter Druck stehen, denken nicht schneller.
[x] verschieben
- [x] Ich möchte nicht unseren Termin nicht verschieben.
[ ] Sie wünschen sich sicherlich ähnlich viel Respekt für Ihre Themen.
[ ] der Vorwurf
[x] die Botschaften
- [x] Versuchen Sie es lieber mit Botschaften über Sie selbst.
[ ] einleiten - [ ] "Ich möchte gern..." leitet viel eher eine konstruktiv Diskussion ein.
[ ] Eine gemeinsame Lösung, mit der alle zufrieden sind, lässt sich nicht immer erreichen.
[ ] Sie ist dann aber die Basis für die Verhandlung von einem Kompromiss in einem anderen Bereich.
[ ] Wurden Sie zu laut, zu hitzig, zu böse und es tut Ihnen jetzt leid?
[ ] zwar: indeed
- [ ] Das sollte Ihnen im Beruf zwar am besten erst gar nicht passieren, wenn die falschen Worte aber erst einmal raus sind, ist hier eine Entschuldigung umso nötiger. (That should not happen in the first place, but once the wrong words are out there, an apology is even more important.)
[x] (ab und zu): from time to time
- [x] Bei der Verteilung der Arbeit im Beruf muss man ab und zu auch mal eine weniger gerechte Aufgabenteilung hinnehmen. (When it comes to the distribution of work at your job, from time to time one has to accept a less fair distribution of tasks.)
[ ] Nutzen Sie die Chance für einen gemeinsamen Austausch, bei dem Sie kleinere Unstimmigkeiten entspannt ansprechen können.
- [ ] Take the chance for a mutual exchange, where you can address minor disagreements in a relaxed manner.
[x] sich unterhalten: to have a conversation
- [x] Können wir uns kurz mal unterhalten?
[x] das Brettspiel
- [x] Das bekannteste Brettspiel ist Schach.
[ ] anwenden -- benutzen
[ ] Es ist schön, dass ihr alle da seid!
[ ] die Bezeichnung -- die Name
[ ] verwunderlich -- wunderbar
[ ] erscheinung -- Das Regen ist eine Erscheinung vom Himmel, dass ich zu Hause bleiben soll, um Deutsch zu lernen, anstatt Frisbee zu spielen.
[ ] erscheinen
[ ] nichtssagend -- unbedeutend, banal, geistlos
[ ] entfällt -- not applicable, no longer required
[x] austeilen, einstecken
- [x] Wer teit Kritik aus, wer steckt Kritk ein?
- [ ] der Bürgermeister teilt Kritik sein Mitarbeiteren aus.
[x] gehören
- [x] Kritik Und Respekt gehören zusammen.
[ ] verschulden
- [ ] etw. verschulden: to be responsible for something
[ ] wütenden -- sauer
- [ ] Bei sehr wütenden Kunden kann auch mein Chef nicht mehr helfen.
[x] erwarten
- [x] Die Leute erwarten, dass ich in meinem Programm humorvoll bin.
[ ] das Mitleid
- [ ] ich habe Mitleid mit dem einen oder anderen Politiker.
[ ] verursachen --- to cause, to raise, to bring about
- [ ] Angst verursachen -- to cause anxiety
[x] Unstimmigkeiten -- discrepancies, disagreements
- [x] Wir spreche bei kleinere Unstimmigkeiten an.
[x] Mitleid:
- [x] Manchmal habe ich Mitleid mit dem einen oder anderen Politiker.

2023-07-30

Der Helfte is Geschaft.

ein Kurs nach dem anderen eins nach dem anderen

kann der Geist so schnell expandieren?

im frühen September

verdauen

die komplette Zeitverschwendung

Wenn du bald nach London gehst, dann hast du jetzt nur noch die Chance. nur noch jetzt

Frontalunterricht

erhalten

der austauch

die sokratische Methode

wenn die Fenster geöffnet sind -- gegenteil wenn die Fenster geöffnet worden sind -- vergangenheit wenn die Fenster geöffnet wurden -- vergangheit

der Lärm

der Lag -- die Verzögerung -- (latency)

der Schwachpunkt

Fetzen n achmachen

der Rückgriff

Plastizität

Neuroplastizität

ganz gut Klapp

irgendwie gleich anwenden -- to appl

ganz gut geklappt

bin und habe als Softwareingenieur gearbeitet

Konsens zu erreichen

übereinstimmen

stimmen uber etwas ein. --> Ich stimme mit dir überein.

links and wordlists

Thingies

"it depends on ... " == "es hängt davon ab.."

Was ist deine Lieblingsessen? Es hängt davon ab, was heute gefällt mir.

the um ... zu construction

It means "in order to".

Ich bin in Berlin angekommen, um Deutsch zu lernen.
I arrived in Berlin in order to learn German.

either or in German is "entweder... oder "

Wir können entweder ins Kino oder ins Restaurant gehen.
We can either go to the cinema or to the restaurant.

the "zu + infinitiv" construction

The zu + infinitiv contruction (or just the "infinitive construction") is used to elide the subject in a sentence that normally contains one. It introduces a subordinate clause. We change the verb into the infinitive form and put "zu" in front of it.

Ich fand es schwer. Ich klettere auf den Berg. becomes Ich fand es schwer, auf den Berg zu klettern.
I found it difficult to climb the mountain.
Er hat vergessen, seiner Freundin etwas zum Geburtstag zu kaufen. I forgot to buy my girlfriend something for her birthday

da-words, wo-words (dazu, darin, danach, daruber, wohin, woher, woraus, woruber...)

Ich habe einen Schlüssel, ich spiele damit. I have a key, I play with it.
Wovon hast du getraümt? What have you dreamed of?
Es gab einen Berg. Ich habe davon getraümt. There was a mountain. I dreamt of it.
Sie ist besser in Feedback geben --> "Sie ist besser darin, Feedback zu geben"

Note that da-words are only for nonliving things. You would say "Es gab einen Mann. Ich habe von ihm getraümt." instead of davon here.

Indirect questions (Indirekte Fragesätze)

Indirect questions are questions that are included within the structure of another sentence. Because indirect questions are dependent clauses (Nebensätze in German), we have to change the position of the verb. Indirect questions follow certain introductory phrases, see the examples below. They often end with a full stop and not a question mark. source

For example, "I do not know what he is doing today." Ich weiß nicht, was er heute mache.

Here are more examples:

Wie lange hast du gestern mit deiner Freundin telefoniert? Ich kann mich nicht erinnern, wie lange ich gestern mit meiner Freundin telefoniert habe. (Note the position of the auxiliary verb!)

If there is no question word, we use "ob" to introduce the indirect question.

Gehst du oft nach Berlin? Do you come often to Berlin? Er fragt mich, ob ich oft nach Berlin gehe. He asked me, whether I come often to Berlin.
Kommt sie morgen? Is she coming tomorrow? Er fragt, ob sie morgen kommt. He asked whether she is coming tomorrow.

Relativsatz

Unregelmasige Verben

Conjunctions

weil

Weil ich heute früh aufgestanden bin, bin ich jetzt sehr müde.
Ich bin sehr müde, weil ich heute früh aufgestanden bin.

obwohl

Obwohl ich total müde war, habe ich trotzdem das Brandenburger Tor besucht.
Trotzdem habe ich das Brandenburger Tor besucht, obwohl ich total müde war.

Präpositionen

Akkusativpräpositionen

betreffend
bis
durch
für
entlang
gegen
ohne
um
wider (wider meinen Willen -- against my will)

Dativpräpositionen

ab
aus
außer
bei
entgegen
entsprechend
gegenüber
gemäß
laut
mit
nach
seit
von
zu

Welcherpräpositionen

an
auf
auf
hinter
in
neben
über
unter
vor
zwischen

Worte

Adjektiv

zurzeit
angstlich
nervig
brav
zufrieden
laut
irritiert
glücklich
traruig
sicher
irritiert
ruhiger

Adverbs

besonders

other stuff

Was für ... (Akk) -- because

Was für einen Beruf hast du?
Was für ein Instrument kannst du spielen?
Was für Musik magst du?

random words

2023-06-26

spüren: to feel
merken: to notice
sofort:
gewesen: past tense of sein
die Ankunft: arrival
ziehen: to pull
schreien: to shout
tief: deep
einatmen (atmen ein): to take a breath
drinnen: inside
drausen: outside
das Shicksal: fate, destiny.
- Das Schicksal mischt die Karten und wir spielen. Fate shuffles the cards and we play.
das Benzin: petrol.
- Ich habe kein Benzin mehr im Tank. I have no more petrol in my tank.
bestimmen: to decide
nutzen, benutzen: to use.
- Take note that nutzen is for abstract things (talent, opportunities) and benutzen is for concrete things (word, car, phone...)
- Ich nutze jede gute Gelegenheit. I use every good opportunity.
- Kann ich deine Gabel benutzen? Can I use your fork?
aufregen: excited.
entspannen: relaxed
zufrieden: pleased

2022-04-17

Ich denke dass die Bank auf Bild 3 mir besser gefallen.

In einige Staat man kann aus dem Brunnen/vom Brunnen trinken.

Homeless person -- Obdachlose

der Obdachlose -n die Obdachlose -n

Sind Sie Obdachlos?

Ein Obdachloser kann auf der Parkbank schlafen.

Ich brauche dieses Wort für wann ich nach Berlin reisen.

Das war ein scherzen Ich macht Spaß

Hotel söllen jeden Tag geöffnet.

Was gibt es in der Umgebung__

Wie sieht es aus: how does it look like there

Hochhaus Man kann bekommt alles.

vier Etagen

Abteilung

die Umgebung in der Umgebung: in the surrounding area in der Nähe

Meine favourite

In der Umgebung gibt es viele Abfallkorben.

Es gibt viele leute, die nicht zu ende essen.

Es gibt Leute, die nicht schlafen. relativ satz, nebensatz

Ich denke dass wir eng wohnen. ---

anschluss: connect Parkanschluss

hätte gern Was hätten Sie gern

benutzen: to use

Der Tisch auf Bild 4 gefallt mir am besten.

Die Person wirft den Abfall nicht weg. (weg wirfen -- trennbares Verben)

Was denken Sie uber die Person von Schreibtisch 3.

Unter uns ist das Restaurant ABER unten ist das restaurant

Hallo Christine,

Hier sind meine Antworten für (zu?) deinen Fragen:

Was für ein ‚Schreibtischtyp‘ sind Sie? Wohin räumen Sie Ihre Gegenstände? Wie richten Sie Ihre Wohnung (oder Traumwohnung) ein?

Mein Schreibtisch ist nur ein Platz für meine Gegenstände; ich glaube meine Schreibtischtyp ist 'a mess'. Normalerweise räume ich meine Gegenstände von auf dem Tisch zu auf dem Boden, und auf dem Boden zu auf dem Tisch! Wenn ich Zeit habe, ich wurde meine Kleidung an den Schrank hangen... aber das ist nicht oft.

Ich hatte gedacht, dass meine Wohnung sehr Minimalist wurde sein. "Ich will in meine Wohnung kein Tisch, keine Stühle--gar nicht", ich hatte gedacht.

Aber wenn ich meine Wohnung hatte eingezogen, ich realisiert dass das nicht praktisch war. Als meine Freunden mich besucht, es gab keinen Platz zum Sitzen, und wir kann nicht Mahjong spielen. Jetzt, ich weiß besser. Man muss in ihren Gedanken flexibel sein.

Was finden Sie beim Einkaufen wichtig? Lieblingskaufzentrum? Welche Sachen? Einkaufstyp? Wie oft? Mit wem? Warum?

Ich liebe kaufe Lebensmittel ein. Ich kann viele Stunden in diese Geschäfte bleiben. Ich gehe fast jeden Tag zum Lebensmittelgeschäft, um meine Mittagessen zu einkaufen--und ich kaufe immer zu viele Essen ein!

Meine Lieblingsaktivität wenn ich nach andere Land reise ist Lebensmittelgeschäft besuchen. Diese Frage sind sehr interessant für mich: Was esst die Leute in diese Land? Wie kauft die Leute ein--schnell oder langsam? Wer kauft was wann?

Kleidung gefällt mir nicht. Wenn ich mit eine schöne Frau für Kleidung einkaufen, es ist vielleicht OK!

Es gibt (present tense) means 'there is' or 'there are'. It is almost always used in the imperfect tense rather than the perfect tense.

The imperfect form of es gibt is es gab. It means 'there was' or 'there were'.

Als meine Mutter jünger war, gab es einen Freibad im Dorf. – When my mother was younger, there was an outdoor pool in the village. Es gab gestern Abend viele Leute auf der Party. – There were lots of people at the party last night.

2023-07-04

(B1.2 Hausaufgaben in Berlin)

Übers Streiten

Es gibt zwei Typen von Streit: ein Streit über die Fakten, und ein Streit über Gefühlen. Man verwechselt oft die zwei, und dann kommen Konflikte.

Zum Beispiel, wenn man sagt "Spar Geld!", vielleich bedeutet das "Wir müssen für ein Haus sparen", oder "Ich fühle mich nervos dass wir zu wenig Geld haben". Wenn man sagt "Warum kommst du sehr spät zuruck?" vielleicht bedeutet das "Ich muss für dich die Tur öffen", oder "Ich habe dich vermisst".

Für die ersten Type von Streit sollt man Logik benutzen: Wie viel Geld sollen wir sparen? Wie Uhr möchtest du mich zuruckkommen? Aber das schadet dem zweiten Type von Streit. Für die zweiten Type von Streit musst man über Gefühle sprechen: Warum fühlst du dich nervos? Ich verstehe dass du mich vermisst, und ich werde mehr Zeit mit dir verbringen.

Die zwei Geschlechte mögen unterschiedliche Methoden. Männer wollen schneller Lösung finden; Frauen wollen erst verstehen. Das wichtigste Ding ist zu wissen, welchen Streit man hat.

2023-07-06

(B1.2 Hausaufgaben in Berlin)

Morgenroutine

Gesund Leben

Annabelle: Gesundheit ist natürlich wichtig. Aber wie soll ich gesund leben, wenn ich immer so viel Stress habe? Ich arbeite den ganzen Tag und danach muss ich noch 1.000 Dinge erledigen. Für Bewegung bleibt da wenig Zeit. Und mittags esse ich meistens in der Kantine. Schreiben Sie nun Ihre Meinung zum Thema (ca. 80 Wörter)

Ich kann natürlich gut Annabelle verstehen. Als ich in Singapur war, habe ich zwei mal pro Woche Frisbee gespielt, und ich habe jede Wochentag ein Workout im Fitnessstudio gemacht. Aber jetzt habe ich in Berlin keine Zeit. Ich muss arbeiten, ich habe Intensivkurs, und ich habe viele neue Freunde, mit denen zu Zeit verbringen. Ich glaube das wichtigste Ding ist, das unsere Altagsleben Gesund sind. Zum Beispiel, man sollst die Treppen benutzen wenn man zum dem Büro geht. Oder man kann stehen während er arbeitet. Ich bewege mich oft während ich denke. Ich glaube, dass man kann auch sich gesund in der Kantine ernähren, wenn man gesündere Auswahlen nehmen.

2023-07-07

Sich entschuldigen. Schreiben Sie eine E-Mail.

Sie besuchen zwei Mal pro Woche einen Yoga-Kurs. Die Kursleiterin, Frau Moser, hat für morgen Abend ein gemeinsames Abendessen für alle Teilnehmer und Teilnehmerinnen organisiert. Sie können zu dem Termin aber nicht kommen.

Schreiben Sie an Frau Moser. Entschuldigen Sie sich höflich und berichten Sie, warum Sie nicht kommen kömmen. Schreiben Sie eine E-Mail (ca. 40 Wörter). Vergessen Sie nicht die Anrede und den Gruß am Schluss.

Liebe Frau Moser,

leide kann ich morgen nicht zu dem Abendessen kommen. Ich habe eine Beschwerde und ich fühle mich ein bissen krank. Mein Nacken tut weh und ich habe Bauchschmerzen. Ich glaube dass, ich mich nur zu erholen brauche. Hoffentlich werdet ihr viel Spaß ohne mich haben!

Mit freundlichen Grüßen, Zhenghong

2023-07-09

possibly useful insight:

Maybe I'll make a separate thread for this one day. Here was my regimen.

1 do everything you can to surround yourself with people who speak better than you do and communicate with them at every possible instance. i was lucky enough to see a girl who was C1 while i was just starting out, but the goal is to chisel away at the 1) psychological barriers preventing you from communicating with others out of the fear of making grammatical/vocab mistakes 2) reliance on your mother tongue. both of these things are very difficult to overcome, especially number 2 for english speakers as there will almost always be outlets to use english instead of german. don't take them, and conversely don't allow german speakers to take them either.

2 VOCAB. the building blocks of language. getting vocab down is a three pronged attack based on the principle of less is more/more is less.

2a. Have a little diary of new words, spreadsheets are okay too. take down new words that you see and write their meanings down with their contexts (optional). this information will be nice to have in some form to look back up at some point.

2b. take these new words and plug them into Anki. Set the Anki settings to either cloze or whatever type you use to actually type the words in. just saying them out loud won't cut it. my settings were 5 new words per day, 20 review words.

without getting too wonky, 5 new words a day plenty for mortal men. even at the fluency level, where i am taking a shitload of new words in at all times, i still commit myself to 5 a day. anything more than that and you will lose them faster than you can take them in, negating the whole process. have those 5 words a day be your bitch by...

2c. writing those five words down on flashcards (or whatever) and using them throughout the day. the more mental gymnastics you have to pull to work those words into a conversation, the better. you have to earn those words through rote memorization by way of creative application. words in anki getting thrown at you again and again need to have life breathed into them by real-world usage, otherwise they have no visceral meaning to the speaker and become useless.

3 have conversations with yourself in the target language. when you read a book, summarize it to yourself later, paragraph by paragraph or page by page if need be (nothing wrong with that!)

4 write as much as you sustainably can every day. if it's a paragraph, fine, if it's a page, perfect.

5 do not use a cell phone to look up words in class at every single possible fucking juncture. this will ruin your learning experience. this is a massive problem with students in the digital era that extends far past learning another language. when a new word comes across, ask your teacher for a definition or consult a dictionary (english-german dictionaries are perfectly fine). teachers in the GI know that they will use hundreds of foreign words a day and it is their job to explain them to their students. this is also a psychological barrier that will need to be overcome: fear of looking stupid by asking questions. don't succumb to it! look as dumb as you actually are(!!), because most people are probably at the same level and will be gawking at you in 6 months time when you have surpassed them, all because you weren't afraid to make mistakes and look dumb!

6 read ALL THE TIME. books are living examples of how language is used throughout time. it is probably my favorite way of collecting new vocab. the GI has a great library of beginner books: USE THEM!!!!

2023-07-10

Monday morning. A bit tired.

Came back from German. Bought some books to learn German. I did Hausaufgaben on a typewriter.

Good meeting with Ben. 2330 -- 0100, after all was said and done. Hopefully I get some answers soon.

2023-07-15

what is the best way to learn a language

surely it must be: lots of writing practice, lots of speaking practice, lots of hearing practice

my main bottleneck now is my lack of vocabulary.

2023-07-17

Welche Orte in einer Stadt sind wichtig für Sie? Was machen Sie dort?

"Dritter Ort" is für mir wichtig. Ein dritter Ort ist nicht ein konkreter Ort, sondern ein abstrakter Typ von Ort.

Das Konzept von dem "dritter Ort" wurde von Ray Oldenberg, einer Americanischer Stadtsoziologe, in seinem Buch "The Great Good Place" erklärt. Der erster Ort ist die Wohnung, der zweiter das Büro, und dann kommt der dritter Ort. Ein dritter Ort ist ein Ort, in dem Leute viele Zeit verbringen; ein Ort, in dem man in Öffentlichkeit entspannt; ein Ort, in dem man neue Menschen kennenlernen kann. Dritter Orte sind (fast) kostenlos, neutral, einladend, freundlich, und angenehm.

Ein paar Beispiele von dritter Orte sind Kneipen, Bibliotheken, Parks, Plätzen, und so weiter. Ein dritter Ort hat für die Gesellschaft eine riesige Bedeutung. Es ist einen Ort für alle Menschen miteinander zu reden. Obwohl jeder Mensch anders ist -- arm oder reich, jung oder alt -- in einem dritter Ort, alle gleichberechtigt sein. Dritte Orte sind deshalb für Zivilgesellschaft, Demokratie, und Zivilcourage wichtig.

(korrigieren)

"Dritter Ort" is für mich wichtig. Ein dritter Ort ist kein konkreter Ort, sondern ein abstrakter Typ von Ort.

Das Konzept von dem "dritter Ort" wurde von Ray Oldenberg, einem Amerikanischen Stadtsoziologen, in seinem Buch "The Great Good Place" erklärt. Der erste Ort ist die Wohnung, der zweite das Büro, und dann kommt der dritte Ort. Ein dritter Ort ist ein Ort, in dem Leute viele Zeit verbringen; ein Ort, wo man in Öffentlichkeit entspannt; ein Ort, in dem man neue Menschen kennenlernen kann. Dritte Orte sind (fast) kostenlos, neutral, einladend, freundlich, und angenehm.

Ein paar Beispiele von dritten Orten sind Kneipen, Bibliotheken, Parks, Plätzen, und so weiter. Ein dritter Ort hat für die Gesellschaft eine riesige Bedeutung. Es ist einen Ort für alle Menschen, um miteinander abzuschalten. Obwohl jeder Mensch anders ist -- arm oder reich, jung oder alt -- in einem dritten Ort sind alle gleichberechtigt. Dritte Orte sind deshalb für Zivilgesellschaft, Demokratie und Zivilcourage wichtig.

[x] spießig: square, uncool
- Die Menschen im Munschen sind spießiger als Berliner.
[x] der Wolkenkratzer: the skyscraper.
- Der Wolkenkratzer ist insgesamt 1,000 meter hoch.
[x] das Nachtleben: the nightlife
- Berlin ist sehr bekannt für das Nachleben.
[ ] erholen: to relax
das Wohngebäude: the residential building
der Plattenbau: a type of building that was built in the DDR
der Vorort: the suburb
[x] die Sehenswürdigkeit: the point of interest
- Eine Sehenswürdigkeit von Binic ist die Kirche Notre-Dame de Bon Voyage.
[x] der Obdachlose: the homeless person
- In den Jahren nach dem Zweiten Weltkrieg gab es in München viele obdachlose Männer.
die Kneipe: the bar
[x] der Lärm: loudness
- Lärm mindert auch die körperliche und geistige Leistungsfähigkeit.
[x] das Angebot: the offer
- Früher haben verschiedene Banken Kredite für Studenten angeboten.
die Aussicht: the view
- Das Fenster bietet eine schöne Aussicht.
der Fußgänger: the pedestrian
das Fahrzeug: the vehicle
die Lage: the location, position
- Wir wohnen in einer guten Lage.
das Schaufenster: the shop window
[x] leise: quiet
- Die leise Stimme des Gewissens kann man auch zum Schweigen bringen.
das Dach: a roof
[x] die Träne: the tear.
- Der Mann ist keine Träne wert.
die Vermutung: the supposition
vermuten: to suppose, to suspect
die Entzundung: the inflammation
echt: real, genuine
- Das Fleicht is echt.
[x] die Geschwindigkeit: the speed
- Die Geschwindigkeit des Lebens in der Stadt ist sehr schnell.
begreifen: verstehen
[x] deswegen: deshalb, daher, darum...
- Nicht wenige Entertainer verfielen deswegen selbst dem Alkohol.
versorgen: to supply, to provide
[x] das Versorgung: the supply of
- In der Stadt gibt es bessere medizinische Versorgung.
[x] der Eindruck: the impression
- Er hat ein Job Interview einen guten Eindruck gemacht.
[x] die Veranstaltung: the event
- Am Christopher Street Day gibt es viele verschiedene Veranstaltungen.
[x] eindeutig: clear, obvious
- In Berlin kann man jeden Tag neues Essen probieren. Das geht in einem Dorf (village) eindeutig nicht.
[x] gegenseitig: miteinander
- Hier sprechen viel leute noch Dialekt, jeder kennt jeden und wir helfen uns auch gegenseitig.
anscheinend: it seems to be
- Landmensch? Anschienend bin ich einer.
[x] einsam: allein
- Man stirbt einsam.
[x] stören: to disturb
- Darf ich dich kurz stören?
[x] überhaupt: absolutely, at all
- Das macht überhaupt keinen Sinn.
- Er war überhaupt keine Hilfe.

2023-07-19

Die Morgendliche Aussicht

Sechs Uhr früh morgens. Die Betrunken stolpern nach Haus zurück während die Arbeitslosen aus euer Häuser stolpern, um einen Job zu finden. Auf der Brücke im Friedrichstrasse Bahnhof, wo ein Obdachsloser schlaf, wurde eine Nackige von einem Voyeur über Glasscherben photographiert. Von meinem Fenster sehe ich alle, und ich denke: obwohl wir in die Gesellschaft alle verschieden sind, scheint die Sonne gleich auf uns alle.

Unser Projekt

Zuerst möchten wir ein Kultur-Treffpunkt am Erdgeschoss bauen, und daruber möchten wir Buros zu Wohnungen umbauen. Da wir diese zwei Projekte kombiniert haben, kosten die Projekte natürlich weniger. Wir sehen für den Umbau und den Treffpunkt insgesamt 10 Millionen Euros ausgeben vor. Die zwei alten Schulen in der Nähe von die Wohnungen werden renoviert werden.

Zweitens möchten wir ein Fahrradweg und Fahrrad-Taxi-Dienstleitung statt der neuen Strassenbahn-Linie bauen. Freiwillige Jugendliche werden alte Leute und Menschen mit Behinderung mit dem Fahrrad-Taxi transportieren.

Meine Stadt

Hallo Eric, mein Schnuckiputzi!

ich freue mich schon darauf, mir meine Stadt zu zeigen. Singapur ist sicher und sehr multikulti. Unser Sprache, "Singlish", wird für dich interessant werden, weil du ein Philologe ist. Es ist eine gut gemischt von Hokkien, Englisch, und Malaysisch.

Obwohl du in den USA Chinesische Essen probieren hast, die Essen hier in Singapur absolut besser ist.

Hoffentlich wirst du Lüst haben und bis bald!

Viele Grüße, Lieu

Wöfur würden Sie es ausgeben?

Es hängt

2023-07-20

jeweils: each, every
- Und dann suchen wir {{jeweils}} irgendeine Kneipe aus.
- Diese Punkte betreffen aussließlich die {{jeweilige}} Stadt.
- Jeder Schuler bekommt jeweils ein Examsheet.
der Pfleger: the nurse
der Frühschicht: the early shift
der Unfall: the accident, the mishap
der Verletzter: the casualty
einliefern: to take someone to the hospital
entfernt: remote, distant
Angestellte
Auszubildenden:
Kunden:
bestellungen:
erwachen: to awake
das Erzieher:
die Müllabführ:
Altenheim:
üblich: usual
der Auszubildende: the apprentice
Bauhof: the builders yard, construction yard
angestellt: to be employed
betrent: to take after
der Verwandter: the relative
das Pferd: the horse
unbedingt: necessary
Sterblichen: the deceased
betrunken: the drunk
stolpern: to stumble
[x] obdachslose: homeless
weltweit:
ausgewählt: to be selected, to be chosen
- Das X hat 40 Städte weltweit ausgewählt.
Gesetzte: the law
bewertet: to review
- Gesetze der jeweiligen Staaten spielen da naturlich eine wichtige Rolle..
Anspruch: claim, right
schließlich:
die Fläche:
Durchschnitt: average
Zugang: access
genannt:
gefährlich: dangerous
Nachteile: disadvantage
gehört: to belong to
dreckig: schmutzig
überzeugen: to convince
begeistet: enthusiastic
genervt: to be annoyed
[x] ausreden: to finish speaking
- Lass mich {{ausreden}}.
[x] vorschlagen: to put forward an idea.
- Welche Lösung {{schlagen}} Sie {{vor}}?
[x] unterbrechen: to interrupt
Rücksicht: consideration (for the poor, disabled, etc.)
[x] regieren: to rule
- Geld regiert die ganze Welt.
gebrauchte: second hand
- die gebrauchte Gitarre
anhalfe
die Einrichtung: the furnishings
abschalten: to turn off
- Beim spazieren gehen kann ich gut abschalten
verzichtet: to quit
[x] stolz: proud
- Ich bin stolz auf dich.
unterhalten: to entertain, to speak.
- Ich unterhalte mich mit dir.
- Er unterhalt seine Gäste.
ubertreiben: to exaggerate.
leisten:
- Ich können mir nicht leisten.
- Das kann ich mir leisten.
[x] vernichten: to annihilate.
- Ich werde dich vernichten.
ausgeben für:
fälschen
stehlen
sich Geld leihen
angeben: to spend
verwenden: benutzen
versäumen
die Schulden
leihen
die Zinzen
die Einnahmen
die Ausgaben
fällig
die mahnung
verlangen
die Rundfunktgebühr
dabheben
das Gehalt
ausreichend

2023-07-21

Ein Überblick von meiner Stadt

In jeder Ecke von Singapur gibt es viele, viele Pflanzen. Die Pflanzen in Deutschland sind ziemlich regelmäsig. Nicht so in Singapur. Eine Blatt der Macrorrhiza ist großer als einen Tisch. Die Blätter der Mimosa schließen, wenn man sie berührt. Im April, gibt es so viele Mangos, dass sie zu Boden fallen und verfaulen. Die ganze Luft duftet von süßen, fruchtigem Wein.

Einige Pflanzen, weil die Sonne auf die gut scheint, oder weil die im gut Erd gepflanzt wurden, wachsen groß und stark. Ihre Blätte wachsen groß und blockieren die Sonne unten. Für andere Pflanzen, die weniger Sonnenlicht bekommen, werden die Blätter kleiner und kleiner und am Ende sterben sie. Es ist traurig, dass die großest Pflanzen keine Zeit haben, um über ihre Brüder und Schwestern nachzudenken.

Jeden tag gibt es so viele Sonnen in Singapur. Die Pflanzen sind von der Sonne geblindet. Die Pflanzen kann nur Sonne sehen, und sie wachseln so schnell sie können, um mehr Sönnen zu bekommen. Sönnen, sönnen, sönnen. Die Pflanzen können gar nicht anderes sehen. Die Pflanzen haben nicht gedacht, was sie eigentlich wollen.

Was wollen sie eigentlich? Um die Welt reisen? Philologie studieren? Eine schöne Frau in einem schmutzigen Kneipe küssen?

Einige Pflanzen haben realisiert, was sie eigentlich wollen. Aber sie haben viel Angst. Sie denken: "Wenn ich nicht so schnell wachseln, wie ich kann, werde ich sterben." Oder, "Ich werde die Welt bereisen, wenn ich Geld habe. Ich kann nicht meinen Job kündigen. Ich muss für eines Haus und meine zukünftigen Kinder Geld sparen" und so weiter.

Und am Ende sterben sie, Kopf voll von Träume.

Ich habe in Europa für ein Paar Jahre gewohnt. Und ich hatte die Glück, viele Europaische Pflanzen kennenzulernen. Mein Wunsch ist, für die Pflanzen in meinem Heimatland, von die Europaische Pflanzen zu lernen.

2023-07-25

treiben --
kräftig -- strong, powerful
das Bein -- the leg

Und da muss man ganz liese in eine Ecke stellen und mucksmäuschenstill sein, damit man nicht auffällt.

Und können sie sich es leisten, langsam aufzustehen oder müssen Sie sich jeden Morgen beeilen?

bitte sprecht mich nicht an Bist du eine leidenschaftliche Person? Wofür was entfacht in dir Leidenschaft

Es wird von den Prüfungsteilnehmenden erwartet, dass sie den hier vorliegenden Wortschatz zumindest rezeptiv beherrschen, das heiß dass diese beim Bearbeiten der geschriebenen Texte verstanden werden müssen.

je nachdem -- it depends
gurken
empfinden -- to feel something
einhalten -- to observe something, or to halt, to stop
enthalten -- contained, covered, to include, to enclose
die Ehre -- the glory

what does that remind me of" -- to segue from topic to topic

"why" questions -- Why did you decide to do that
what do you do, why do you do it?
laughing is a habit that gets better
mirroring -- last one to three words

A1: 650 A2: 1300 B1: 2400

2023-07-26

[x] überfordern - to be overwhelmed
[x] unabhängig - independent
[x] vertretung - replacement
[x] heraushängen - to hang out

A2 Wortliste (1300 Worte)

[x] die Ampel:
- Wenn die Ampel grün wird kannst du die Strasse überqueren.
[x] anbieten
- Kann ich dich etwas zu trinken anbieten?
[x] aufpassen
- Tut mir leid. Da habe ich wohl nicht aufgepasst.
[x] aufräumen
- Vor meinem Urlaub muss ich unbedingt noch meinen Schreibtisch aufräumen.
[x] aufregen
- Ganz ruhig! Bitte regen Sie sich nicht auf.
- Heute hast du eine Prüfung. Bist du schon aufgeregt?
- Die Musikgruppe zu erleben war sehr aufregend.
[x] die Ausbildung
- Ich habe meine Ausbildung abgeschlossen.
[x] der Ausflug
- Am Wochenende machen wir mit Freunden einen Ausflug nach Heidelberg.
[x] sich ausruhen
- Sie ruhen sich von der Arbeit aus.
[x] außerdem
- Der Film war langweilig, und außerdem haben die Schauspieler schlecht gespielt.
[x] außerhalb
- Wir wohnen außerhalb des Waldes.
[x] aussteigen
- Diese ist meine Haltestelle. Ich steige hier aus.
[x] die Ausstellung
- 20.000 Eintrittskarten waren bereits vor Beginn der Ausstellung verkauft worden
[x] beeilen
- Wir müssen uns beeilen. Sonst verpassen wir den Zug.
[x] begründen
- Bitte begründen Sie Ihre Meinung.
[ ] beraten
- Unser Personal berät Sie gern, wenn Sie Fragen haben.
[x] berichten
- Alle Zeitungen haben über den Unfall berichtet.
[x] der Bescheid
- Den endgültigen Bescheid erhalten Sie in etwa vier Wochen.
[x] (sich) beschweren
- Wo können wir uns beschweren (to complain)?
[x] besonders
- Es ist besonders gut, dass du Big Dick Energy hast.
[x] bestätigen
- Die Firma hat mir den Termin schriftlich bestätigt.
- Ich kann bestätigen, dass er hier war.
[x] (sich) bewerben
- Ich habe mich um diese Stelle beworben.
- Ich habe mich als Kellner beworben.
[ ] bitten
- Darf ich Sie bitten, kurz zuzuhören?
[x] das Blatt
- Das Blatt der Macrorrhiza ist so groß wie ein Tisch.
[x] die Bohne
- Ich esse gern Bohnen.
[ ] braten
- Das Fleisch muss zehn Minuten braten.
- Heute gibt es gebratenen Fisch.
[ ] buchen
- Ich habe für morgen einen Flug nach Rom gebucht (to book).
[x] darauf
- Darauf fällt mir keine Antwort ein.
[x] darüber
- Darüber spreche ich nicht gern.
[x] dabei
- Hast du einen Stift dabei?
[ ] dafür
- Ich bin dafür.
[ ] dagegen
- Ich bin dagegen.
[x] damals
- Mit 15 wollte ich gerne Klavier spielen lernen. Aber damals hatte ich kein Geld dafür.
[x] deutlich
- Schreiben Sie bitte deutlich (neatly)!
[x] dringend
- Ich muss dich dringend (eilig) sprechen.
- Ich erwarte einen dringenden Anruf.
[x] drüben
- Dort drüben ist die Haltestelle.
[x] drücken
- Sie brauchen nur auf den Knopf zu drücken.
[x] drucken
- Bei welcher Firma haben Sie diese Prospekte drucken lassen?
[x] dünn
- Mein Sohn ist sehr dünn (thin). Er isst zu wenig.
- Warum ziehst du nur so einen dünnen Mantel an? Es ist kalt draußen.
[x] eintragen - Tragen Sie sich bitte in die Liste ein.
[x] einverstanden
- Bist du einverstanden, wenn wir bald nach Hause gehen?
[x] einzeln
- Dieses Mineralwasser gibt es nur als Sechserpack, nicht einzeln.
[x] einziehen
- Die Wohnung ist fertig. Wir können nächste Woche einziehen.
[x] erkältet
- Ich bin erkältet, daher kann ich heute nicht kommen.
[x] die Erlaubnis
- Sie hätten ihn vorher um Erlaubnis bitten müssen.
[x] die Ermäßigung
- Es gibt eine Ermäßigung für Kinder und Senioren.
[x] das Fach
- Die Handtücher sind im obersten Fach links.
- Welches Fach magst du in der Schule am liebsten?
[x] die Abfahrt
- Es bleiben uns noch zwanzig Minuten bis zur Abfahrt.
[x] faul
- Mein Sohn war am Anfang in der Schule sehr faul. Heute lernte er sehr fließig.
[x] sich freuen
- Es freut mich, dass alles so gut geklappt hat.
- Ich habe mich über Ihr Geschenk sehr gefreut.
[x] froh
- Ich bin froh, dass alles so gut geklappt hat.
[x] der Führerschein
- Wenn Sie betrunken Auto fahren, verlieren Sie Ihren Führerschein (driving license).
[x] furchtbar
- Gestern Abend gab es ein Gewitter. Es hat furchtbar (terribly) geblitzt und gedonnert.
- Das war mir furchtbar (terribly) unangenehm.
[x] die Gabel
[x] das Gehalt
[x] die Geldbörse
- [x] Ich habe nur Kleingeld in meiner Geldbörse.
[ ] das Gerät
[ ] das Gericht
[x] das Geschirr
[ ] das Gewitter
[ ] die Grippe
[x] gültig sein
[x] der Hals
[x] die Haltestelle
[ ] das Heft
[ ] die Heizung
[ ] der Herd
[ ] herstellen
[ ] herunterladen
[ ] holen
[ ] husten
[ ] die Jugendherberge
[ ] die Kasse
[ ] die Kenntnisse
[ ] die Kette
[ ] klappen
[ ] die Kreuzung
[x] (sich) kümmern
- [x] Die Polizei kümmert sich um die Sicherheit.
[x] kündigen
- [x] Ich war mit meiner Stelle (Arbeitsplatz) nicht zufrieden und habe gekündigt (resigned).
[ ] leer
[x] liefern
- [x] Wir liefern (to deliver) Ihnen das Regal (the shelf) direkt ins Haus.
[ ] die Lüge
[x] der Magen
- [x] Mit tut der magen (bauch) weh. Ich habe Magenschmerzen (Bauchschmerzen).
[x] malen
- [x] Ich male (paint) die Wand rot mit meinem Blut.
[ ] die Mannschaft
[ ] der Mantel
[x] die Menge
- [x] Diesen Preis gibt es nur, wenn Sie große Mengen (amount) kaufen.
- [ ] Ich kenne schon eine Menge Leute in der Stadt.
- [ ] Das Auto hat bestimmt eine Menge geld gekostet.
[ ] die Mütze
[x] nass
- [x] Es hat schrecklich geregnet. Ich bin ganz nass (feucht) geworden.
- [x] Geh nicht mit den nassen (feucht) Schuhen ins Wohnzimmer.
[ ] nebenan
[x] neblig
[ ] nirgends
[x] das Pferd
[x] raten
- [x] Ich möchte billig Urlaub machen. Was raten (to advise) Sie mir?
- [x] Rate (to guess) mal, wen ich heute getroffen habe! Deinen Lehrer.
[ ] das Rätsel
[ ] rechnen
[ ] der Reifen
[x] die Reihe
- [x] In {{jeder Reihe}} (every row) saß eine Person.
[ ] die Reinigung
[ ] reiten
[ ] sammeln
[ ] schädlich
[ ] schaffen
[ ] der Schalter
[ ] die Schere
[ ] das Schiff
[ ] das Schild
[ ] schimpfen
[ ] schlimm
[ ] schrecklich
[ ] schriftlich
[x] schwach
[x] die Seife
[ ] die Sendung
[ ] speichern
[ ] stattfinden
[ ] die Stelle
[ ] der Stiefel
[ ] stören
[ ] streng
[ ] sympathisch
[ ] die Tafel
[x] teilnehmen
- [x] Ich werde morgen am Frisbee-Spiel teilnehmen.
[ ] übernachten
[ ] übersetzen
[x] die Übersetzung
[x] unbedingt
- [x] Ich muss unbedingt (absolutely) mit dir sprechen.
[ ] der Unfall
[ ] (sich) unterhalten
[ ] die Unterkunft
[ ] unternehmen
[ ] untersuchen
[x] verabredet sein
[x] der Verein
[ ] vereinbaren
[x] verletzt
[ ] verliebt
[ ] verlieren
[ ] verreisen
[ ] verschieben
[ ] versuchen
[x] der Vertrag
[ ] vorher
[ ] vorn(e)
- Bitte vorn(e) einsteigen
[ ] vorsichtig
[ ] vorstellen
[ ] vorwärts
[ ] wach
[ ] wecken
[ ] wegnehmen
[x] der Wettbewerb
[ ] die Zahl
[ ] zeichnen
[ ] die Zeitschrift
[ ] der Zettel
[ ] das Zeugnis
[ ] ausgezogen
[ ] zufrieden
[ ] zuhören
[ ] zuletzt
[ ] zumachen

A1 Wortliste (650 Worte)

[x] abfahren: to depart
[x] abgeben
[x] abholen
[x] der Absender
[x] anbieten
[x] anklicken
[x] ankreuzen
[x] die Anrede
[x] der Anrufbeantworter
[x] die Ansage
[x] der Anschluss
die Anzeige
[x] sich anziehen
[x] aufhören
[x] die Auskunft
[x] aussteigen: to get off (a stop)
[x] der Ausweis
[x] (sich) ausziehen
[x] besetzt: occupied
- die nummer ist immer besetzt.
- der platz ist besetzt.
[x] besichtigen: to visit (a place)
- Sehenswürdigkeiten besichtigen.
[x] die Birne -- the pear
[x] der Bleistift -- the pencil
der Blick -- (look, glance). überraschter Blick, im Blick auf ..
[x] böse -- vicious, wicked, angry...
- Ich bin böse auf mich.
- Ich habe dein Buch vergessen. Hoffentlich bist du mir nicht böse.
[x] daneben
- Du kennst doch die Post. Daneben ist die Bank.
[x] drücken - to squeeze, to push, to press
- Drück hier, dann geht der Computer an.
die Durchsage: die Anssage
[x] eilig: urgent
- Ich habs nicht eilig. Und du?
[x] einsteigen: to enter, to step in, to board
- Schnell, steig ein, der Zug fährt gleich.
[x] der Eintritt
der Empfänger: the recipient
- Empfänger unbekannt.
[x] das Ergebnis
[x] erlauben: allowed
- Rauchen ist hier nicht erlaubt.
[x] erzählen: to tell something, to narrate
- Lügen erzählen: to tell lies
- Ich muss Ihnen davon erzählen, um es mir vom Herzen zu reden.
- Es schadet nie, eine gute Geschichte zweimal zu erzählen.
- Die Himmel erzählen die Ehre Gottes.
[x] fehlen: to lack, to miss
- Mir fehlen die Worte. Words fail me.
- Entschuldigung für mein Fehlen.
[x] das Gepäck: the luggage
- Lassen Sie ihr Gepäck nicht unbeaufsichtigt!
- Wollen Sie Ihr Gepäck mitnehmen?
[x] geradeaus: ahead, straight on
- Gehen sie geradeaus.
[x] das Gewicht: the weight, the severity, the importance
- etwas Gewicht verleihen
- an Gewicht zunehmen, an Gewicht verlieren
- sein Gewicht in Gold wert.
- Ist dies ein Argument von Gewicht?
[x] gültig:
- Der Pass ist nicht mehr gültig.
[x] die Haltestelle:
- An der nächsten Haltestelle müssen Sie aussteigen.
[x] hell
- Im Sommer ist es bis 21 Uhr hell.
[x] der Herd
- In der neuen Küche fehlt noch der Herd.
[x] holen:
- Ich hole zwei Flaschen Wasser aus der Küche.
[x] die Kasse
- Zahlen Sie bitte an der Kasse.
[x] kriegen: verdienen?
- Ich kriege 15 Euro in der Stunde für meine Arbeit.
[x] kümmern sich um:
- Jede Mutter kümmert sich um ihre kleinen Kinder.
[x] ledig:
- Sind Sie verheiratet? – Nein. Ledig.
[x] mitnehmen
[x] das Öl
[x] die Rechnung
[x] das Schild, -er
der Schluss
- Ich muss jetzt Schluss machen
- Zum Schluss gibt er uns allen die Hand.
[x] zu sein: to be closed
- Die Tür ist zu.
[x] sofort: immediately
[x] übernachten
[x] überweisen
[x] umziehen
[x] unterschreiben
[x] die Unterschrift
[x] der Verein
[x] weiblich
[x] weiter
- Der Bus fährt nicht weiter.
[x] der Zoll

625 words

der Schauspieler -- actor
das Adjektiv -- adjective
der Erwachsene -- adult
die Luft -- the air
der Flughafen-- the airport
lebendig -- alive
das Tier-- the animal
die Wohnung -- apartment
der Apfel -- apple
April -- April
der Arm -- arm
die Armee -- army
die Kunst -- art
der Kunstler -- artist
der Überfall -- the attack
August -- August
der Author -- the author
das Kleinkind -- the baby
der Rücken -- the back (body)
hinter -- back (direction)
schlecht -- bad
die Tasche -- the bag
das Ball -- the ball
die Banane -- the banana
die Band -- the band
die Bank -- the bank
die Kneipe -- the bar
das Badezimmer -- the bathroom
der Strand -- the beach
der Bart -- the beard
schlagen -- beat (someone)
schön -- beautiful
das Bett -- bed
das Schlafzimmer -- bedroom
das Rindfleich -- the beef
das Bier -- the beer
biegen -- bend
das Getränk -- the beverage
das Fahrrad -- bicycle
groß -- big, large
die Rechnung -- the bill
billion
der Vogel -- the bird
schwarz -- black
blind -- blind
das Blut -- blood
blau -- blue
das Boot -- boat
der Körper-- body
der Knochen-- bone
-- book
-- bottle
-- bottom
-- the box
-- the boy
-- the brain
-- the bread
-- to break
-- breakfast
-- the bridge
-- the brother
-- brown
-- to build
-- the building
-- to burn
-- the bus
-- to buy
-- the cake
-- to call
-- the camera
-- the camp
-- the car
-- the card
-- to carry
-- the cat
-- to catch
-- the ceiling
das Handy -- the cellphone
-- centimeter
-- the chair
-- cheap
-- the cheese
-- the chicken
-- the child
-- the circle
-- the city
-- the clay
sauber -- clean (adj.)
putzen -- to clean
die Uhr -- the clock
schlossen -- to close
die Kleidung -- the clothing
-- the club
-- the coat
-- the coffee
-- cold
-- the colour
-- the computer
-- the consonant
-- the contract
-- to cook
-- cool
-- the copper
-- the corn
-- the corner
zählen -- to count
das Land -- the country
das Gericht-- the court
-- the cow
die Gedränge -- the crowd
-- to cry
die Tasse-- the cup
gebogen-- curved
schneiden -- to cut
tanzen -- to dance
dunkel -- dark
das Datum -- the date
die Tochter -- the daughter
der Tag -- the day
der Toter, die Toten -- the dead
taub -- deaf
der Tod -- the death
der Dezember -- December
tief -- deep
das Diamant -- the diamond
sterben -- to die
graben -- to dig
das Abendessen -- the dinner
die Richtung -- the direction
schmutzig -- dirty
die Krankheit -- the disease
-- the doctor
-- the dog
-- the dollar
-- the door
-- the dot
-- down
-- to draw
-- the dream
-- the dress
-- to drink
-- to drive
-- the drug
trocken-- dry
-- the dust
-- the ear
-- the Earth
-- the east
-- to eat
-- the edge
-- the egg
-- the election
-- the electronics
-- the energy
-- the engine
-- the evening
-- the exercise
teuer -- expensive
-- to explode
-- the eye
-- the face
-- the autumn
-- the family
-- famous
-- the electric fan
-- the sports fan
-- the farm
-- fast
-- the father

Path UBI

2023-09-10T00:00:00Z

The case for PATH

Vision: to bring value into the world. Mission: to give every creator a basic income of $1,000 a month so they can create without concern.

Imagine a toy economy where taxes accrue to the treasury and are used to fund a UBI of sorts.

There is a token2022 called PATH.
Each transaction involving PATH pays a 0.1% transfer tax, which accrues to the treasury.
The treasury gives out PATH daily equally to all verified human accounts (leave the proof-of-personhood aside for now), capped at X PATH/person/day.

Note we couldn't do this with USDC or any other currency that we do not control, as USDC does not allow for a tax-on-transfer. The challenge then is to incentivise the velocity of PATH, by creating an ecosystem (like the real economy) of goods and services.

Tons of ideas of how we do that were expressed above, but at a 0.1% transfer rate, the total velocity of money has to be 1000x the amount distributed per day. If you want to give out 1,000 PATH per day, you need 1M PATH moving in and out of the system.

The pie-in-the-sky case is that there's a 10M, 100M PATH of volume moving around everyday worth of goods and services transacted. 10M == 10,000 PATH per day, 100M == 100,000 PATH per day.

Sources and sinks

For this to work the treasury must either keep growing or be revenue neutral.

Inflows:

Investment or business income
0.1% transfer tax

Outflows:

Basic income

Inflows >= outflows.

Suggestions for inflows:

Anyone can suggest an business idea to the Pathfinder treasury DAO
- e.g. "ChatGPT on Telegram" -- get a grant on completion and in return revenue share with the Pathfinders DAO
- e.g. "SUPERHOT wallet on Telegram" -- a superhot wallet that's 1-of-2 custodial (although ideally making it noncustodial would be amazing), that comes with quick shortcuts e.g. "/transfer 1 SOL to 0xfp.sol", "/swap 1 SOL to USDC", "/withdraw all to 0xfp.sol". The use case would be to reduce the friction of onboarding all my friends from Telegram (can start sending our employees PATH!)
  - literally a tiplink-esque thing would be nice, /send @celineli 1 PATH
  - Which Pathfinder/Pathfinder creature is best for wallet? "Elio" for ChatGPT, "X" for Wallet?
  - FYI: Bots are now able to use collectible usernames — including links without the -bot suffix.
- I think this alone is a huge idea Stripe Crypto onramp from Telegram maybe..
But to receive a basic income one has to:
- be a real person;
- be a Pathfinder;
- be near-unanimously voted in by the other Pathfinders (which means they must have demonstrated a track record of value)
Number of Pathfinders is (for now) hard capped at 10,000, but obviously Editions 2, 3, 4 etc. are possible. I would keep it as small as possible though at the start.

Synergy with Pathfinders

Strategy: Use the Pathfinders NFT project to bootstrap the Pathfinders economy. Build goods and services that can be purchased at a discount (or exclusively) with PATH.

As the economy grows and the treasury accumulates, each Pathfinder NFT will be worth more and more money because they hold the keys to the kingdom: only Pathfinder NFT holders are eligible to receive a basic income.

We bootstrap the economy by paying all contributors to the Pathfinder project with PATH. True believers can buy PATH and start buying goods and services (e.g. art, compute, GPT-time, plushies, stickers...) with PATH.

The goal for us is to make stuff that people want to pay for and take PATH as payment. One example is to convert the mint proceeds from SOL to PATH, and to charge people PATH for any future live-service content.

Pathfinders is an IP (universe), a community, and now an economy. There's a reason for the community to stick together and consume goods and services from one another: we get richer together.

In the future businesses will be incentivised to "hook in" to the Pathfinders community (ergo accept PATH) to benefit from the goodwill, the free marketing, and also they might be able to charge a slight premium because believers want to use PATH (??? questionable)

PATH Visa card? LOL -- how would this work

Things to look out for

At the beginning we must avoid at all costs people who sign up to get the UBI and contribute nothing to the economy. This is the problem with all past UBI projects: they didn't think about the sinks, only the sources. Onboarding people too fast is a death knell.

In the future if the economy is sufficiently closed (you can buy everything you need to eat, rent a Pathfinders capsule hotel), or if we become "post-scarcity" (we make so much money from our Pathfinders businesses that we can give that surplus away), then this would be no problem, but that's still a long ways away.

I would position it not as a universal basic income (because there are 8 billion people out there), but a global collective of creators (not creatives -- programmers, farmers, restauranteurs) supporting one another.

Closing thoughts

I will say that if we give 10,000 people $1,000 a month, that's an outflow of $10M/month. That's an incredible amount. Can the treasury really make it back? That defies expectations. Literally incredible. They should feel like the luckiest 10,000 people on Earth. Lol.

on education

2023-07-15T00:00:00Z

Brian Caplan's "The Case against Education" -- education is 1/3rd signaling, 1/3rd ability bias, 1/3rd human capital growth

How to increase brain plasticity for language learning

2023-06-27T00:00:00Z

Critical Periods in Speech Perception

![[Screenshot_2023-06-27-07-17-56-532_org.mozilla.firefox.jpg]]

REOPENING CRITICAL PERIODS

A continuing question is whether the age-related changes in phonetic perception involve a full loss of sensitivity. Those consonant contrasts that show the most robust evidence of perceptual attunement in infancy—such as the English /r/-/l/ and the Hindi /d/-/D/—are the same ones that show the most resistance to training in adulthood. With systematic training, however, significant improvement on discrimination of nonnative contrasts can be induced (e.g., Bradlow et al. 1997, Lively et al. 1994, McCandliss et al. 2002), particularly when feedback is provided (Lively et al. 1994) or the training progresses from the easiest to the most difficult tokens (Ingvalson et al. 2012). Although training seldom leads to improvements to the level of native speakers (e.g., Bradlow et al. 1997), there are large individual differences in performance following training (McCandliss et al. 2002), with one predictor of success the thickness of white matter tracts in Heschl's gyrus in the left hemisphere (Golestani et al. 2007).

Animal models now indicate that the mature brain is intrinsically plastic and actively stabilized by a variety of brake-like factors in adult life (see Table 1). As a result, it is possible to reopen CP levels of plasticity by judicious lifting of these brakes. Two potent pathways by which to do this involve focused attention and epigenetics (turning on or off genes), both of which can reestablish the E-I balance necessary for opening plasticity (see sidebar for further details). In the past, we would have explained the improvements in nonnative speech perception in adults as evidence that the CP had never fully closed and/or that there was thus some residual plasticity. Today we can investigate an equally plausible explanation: The training regimens that are most effective work by activating biological processes that remove molecular brakes.

The action of neuromodulators (such as the serotonin and nicotinic acetylcholine receptor signaling onto particular GABA neurons) acutely regulates E-I balance (Lee et al. 2010), effectively disinhibiting the local circuitry to enable adult learning (Brown et al. 2012, Donato et al. 2013, Letzkus et al. 2011). In adult barn owls, active hunting can extend CP plasticity (Bergan et al. 2005). Video games are particularly potent in engaging attentional mechanisms, which can be leveraged for enhanced learning (Bavelier & Davidson 2013). Highly engaging learning situations similarly may be more effective in improving speech perception in adults.

Stimulation of the basal forebrain or vagus nerve raises neuromodulatory tone to produce CP plasticity in adult A1 (Engineer et al. 2011, Kilgard & Merzenich 1998) sufficient to correct a rodent model of tinnitus. In V1, the Lynx1 protein was further identified as a brake-like factor that actively suppresses adult plasticity (see sidebar) (Morishita et al. 2010). Genetic removal of Lynx1 or acetylcholinesterase inhibitor treatment of adult wild-type animals restores CP plasticity, enabling recovery from amblyopia in adulthood. Likewise, phase 2 clinical trials with SRIs are proceeding for the treatment of amblyopia, as these drugs were found to rescue amblyopia in a rodent model by acutely perturbing E-I balance, perhaps epigenetically (Maya Vetencourt et al. 2008, 2011). Interestingly, contemplative practices to focus attention may also achieve plasticity enhancement noninvasively by engaging neuromodulatory systems (Slagter et al. 2011) as well as by producing epigenetic changes (Kaliman et al. 2014). This is another area for future study.

Epigenetic signatures such as DNA methylation, histone acetylation, and phosphorylation accompany the closure of CPs (Putignano et al. 2007). Thus, treating adult rodents with HDAC inhibitors such as valproic acid alters preference to paired acoustic stimuli (Yang et al. 2012) or reopens plasticity and rescues amblyopia in V1 (Silingardi et al. 2010). The same valproic acid treatment of healthy young adult humans enables the acquisition of absolute pitch, again when trained concurrently (Gervain et al. 2013). Elimination of inhibitory synapses is a major component of adult visual plasticity (Takesian & Hensch 2013, van Versendaal et al. 2012).

As noted at the beginning of this review, language is one of the few complex cognitive systems for which CPs exist. This may have implications for when and whether training of speech sound discrimination can improve language use. To date, the improvements reported at the level of speech sound (phonetic) discrimination seldom generalize to new syllables or new speakers (Ingvalson et al. 2011). Moreover, even when nonnative phonetic sensitivities improve with training, deficits still exist in how well the retrained categories facilitate the recognition of words in continuous speech (e.g., Pallier et al. 1997). In animal model work, prolonged sensory deprivation (such as dark rearing or white noise in adulthood) has been shown to change the E-I balance and reopen plasticity (Duffy & Mitchell 2013, He et al. 2006, Zhou et al. 2011). It remains to be determined whether comparable manipulations would similarly reopen CPs in language acquisition, but it is a theoretical possibility. Moreover, since input at all levels of the language system—not just phonetic perception—would be removed, perhaps higher-level use of speech perception sensitivities would also improve.

CONCLUSIONS AND FUTURE DIRECTIONS

In summary, infants begin life with a set of perceptual biases, learning mechanisms, and neural systems that orients them to language, provides an initial foundation for categorization, and allows them to learn the properties of the native language. Infants become attuned to these properties in a sequential fashion, beginning with the most global prosodic characteristics of the native language and eventually reaching to detailed phonetic analysis. In at least some cases, the timing of attunement appears to be under maturational control, suggesting the operation of a critical or sensitive period. The perceptual attunement to native language characteristics supports acquisition of the relevant structures, for example, of cues to word order or of word-meaning mappings. Moreover, attunement of the properties at each tier enables more focused attention to the next level of specificity.

It is often claimed that because there is lifetime plasticity, CPs are moot—but that misses the point. There are biologically verifiable CPs beyond which the brain works hard to maintain stability. By understanding the processes that function to maintain stability (molecular brakes), we may figure out how to remove those brakes and reopen plasticity. The opening and closing of a CP is like opening and then locking a door. Once a key is turned or a combination set, the door is locked. However, if one has the key or knows the combination, the door can be unlocked. The exciting work in neurobiology over the past 15 years has shed light on how that unlocking is done (Takesian & Hensch 2013).

New advances have brought within reach a more objective mechanistic definition of CP timing as a consequence of maturational state. Future directions include the combination of animal models with human work, as illustrated above. For example, despite the long-postulated parallels of birdsong and human language acquisition (Doupe & Kuhl 1999), only recently has mechanistic work been done with respect to CPs (Balmer et al. 2009). In the barn owl system, incremental training is effective in producing cumulative large changes in auditory maps (Linkenhoker & Knudsen 2002).

In addition, one can now carefully examine human populations with molecular markers in mind (see Table 1). Methods borrowed from experimental neuroscience, such as serendipitous pharmacology or epigenetic profiling, can be applied to track the trajectory of human language acquisition. Notably, many mental illnesses that include language impairment—such as schizophrenia and autism—are of neurodevelopmental origin and share defects in the triggers and brakes that normally regulate CP timing (Gogolla et al. 2009, Insel 2010, Rubenstein & Merzenich 2003). Interventions to restore CP mistiming are starting to yield promising therapeutic strategies in animal models (Gogolla et al. 2014).

In summary, in this article we have reviewed recent work on CPs in speech perception development within the mechanistic framework of understanding how sensory CPs work at a biological level. We are hopeful that by bringing new insights from neuroscience to one of the longest-standing debates in the language acquisition field, we can move beyond old strictures to better position scientists and clinicians across fields to gain a deeper understanding of how language development unfolds and to implement more effective interventions when development goes awry.

Georgism in Singapore

2023-06-24T00:00:00Z

https://progressandpoverty.substack.com/p/singapore-economic-prosperity-through

https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=1006&context=soe_research

Guerilla Gardening

2023-06-04T00:00:00Z

Idea of guerilla gardening is to plant shit wherever without soliciting the permission of the government or city council or whatever. Just plant stuff, let it grow, eat some papaya. You'd be surprised at how much shit grows in a small space.

Joy

2023-05-15T00:00:00Z

Eight years ago I read Anna Karenina in a cold and dark storeroom. I don't remember the plot exactly now, but I remember how I felt. I remember it gripped me from start to end. I remember viscerally feeling Anna's aching passion and her pain, and when I read that final, climactic scene, it felt like a train had hit me, too. I read that part a dozen times, reliving it again and again before I had to put the book down. I felt a sickening sense of loss. My heart pounded hard and my chest felt incredibly tight, stretched taut almost to breaking underneath my admin tee. Jaye and Benny were only inches away from me. But in that moment I felt incredibly distant, my mind and body transported to a time long ago, a place far away. Reading that book gave me joy.

I took many modules in college, and enjoyed all of them. Learning new things gave me joy. I enjoyed working through particularly tricky proofs in Logic, Analysis, and Game Theory. I enjoyed very much thinking through Berlin and Nozick. I worked hardest for PolSoc and my thesis. I remember lying on the couch in the PPE room with my feet up and my eyes closed, thinking very hard in my repose. I'd fill the whiteboard with papers, summaries, regression equations. Occasionally I'd sneak a peek at Julia Stadlmann--hard at work with some derivation or other--and sometimes she'd catch me and grace me with a smile and giggle. That gave me joy. I enjoyed parking the disparate readings into different camps, finding methodological errors in the papers, and finally synthesising everything into a theory of my own. I came to every insight with nothing but the Feynman method, and that exercise of my mental faculties gave me joy.

I talked to so many people about my thesis, who lavished me with help I didn't deserve. I was in Tak Huen's attic room. I was in the common room of 59 with Filip, Nina, and Martin. I was in Bassel's garden, talking until the sun went down. I was in Nuffield quad and Andy's office. I was in my room writing code until the sunrise. All of that was pain, uncertainty, and stress. But at the end, when the results aligned almost exactly with my predictions, I felt that I had discovered some truth about the world. A contingent, a posteriori, wishy-washy sort of truth, but a truth nonetheless. That gave me joy. And I felt unadulterated joy when Celine and I received our results atop Kühlenbronn, with the sun in our eyes and the cool mountain air in our lungs.

Last year I started playing frisbee. Why do I play it? It keeps me fit and lets me meet new people, but those reasons are only incidental. I play for the brute joy of running down the field as quickly as I can and jumping as high as I can. That raw exercise of my physical capacities is joyful to me. There is beauty in the game too, which I'm starting to appreciate. There is beauty when a play is executed nicely, when the disc moves swiftly from strike to strike and then put nicely and cleanly to score in the endzone. There is beauty when someone reads the disc so well that he grabs it out perfectly from two defenders. Those things give me joy.

Some time ago I had my heart broken. We spent one intimate and vulnerable night together. That gave me joy. Then I spent many sleepless nights alone feeling pain, anguish, and overall very sorry for myself. But I felt like I hadn't felt in years--a full-bodied, hot-blooded grief--and that gave me joy too.

On Tiktok and Instagram Reels

2023-05-14T00:00:00Z

Last night a friend showed me her Instagram DMs. Too many of her conversations were reels back and forth punctuated by a few words of filler reaction ("lol", "HAHAHAHAHA", "very cool", etc). How could it be otherwise? There are no scintillating rejoinders to a tiktok where someone is pulling funny faces while making a very large portion of food.

It is good to share Internet stuff with one's friends. But these tiktoks are so vacuous that they terminate thought instead of sparking discussion. They are the 2020s version of an MSN nudge or a Facebook poke: they contain the form of a message, but have none of the substance, like biting into a xiaolongbao and getting only air.

A conversation of reels is a poor substitute for an actual one. It's become far too easy to ping-pong Tiktoks back and forth; far too easy to shirk the work of having real conversations with the people you love.

Using technology to aid language learning

2023-04-23T00:00:00Z

Some scattered thoughts on how we can use technology to build something 100x better for language learning

ChatGPT can act as a native speaker

ChatGPT can give you exercises

Here is ChatGPT

Prompt:

German Konjunctiv II practice

I would like to practice the German Konjunctiv II mood, and I would like you to be my helpful and knowledgeable German tutor. I would like you to give me a sentence in English, and grade me on my translation. The sentences you give me should use the Konjunctiv II mood.

German Reflexive Verb Practice

I am looking to practice German reflexive verbs. I would like you to give me a reflexive verb in German, with an example English sentence using the verb. I should translate the sentence into German correctly.

An example:

Reflexive verb: sich freuen (to be happy)
English sentence: I am happy because I have a party on the weekend.

You should wait for me to give you my attempt at translating the sentence to English before you give me the correct German translation.

A proper dictionary extension and flashcard capturing extension

Idea: something that allows you to highlight any word or phrase, and it automatically looks up dict.cc

also it should give you the option to translate it automatically and capture it into a flashcard deck. maybe even do text-to-speech

snip video fragments and put them into flashcards??

idk

AI-generated flashcards

Improving the site (2023)

2023-04-08T00:00:00Z

Introduction

I have been making new posts regularly, but haven't changed either the design or the backend of the website since 2020. Time flies! I wanted to do it back in 2022 but 2022 was a very busy year for me. Thankfully things are less hectic now.

Removed Github Pages Actions, moved to Vercel

It was taking upwards of 20 minutes for any change to reflect on my website after I pushed the changes. It turned out that Github Actions save something called an artifact, which took a long time (~10 minutes!) given my many huge files in the repo.

Serving purely static files shouldn't take anywhere near that long. I switched to Vercel instead, now it builds in seconds. I'm still using GitHub Actions to run eleventy build and encryption (see next section)

Add encryption for private posts

I used to keep private posts up, but events transpired that made this no longer a good idea. I took them down for over 2 years but was always looking for a way to keep them up with password protection. (see my 2022 post). I'd been thinking of a way to do client-side encryption. Two months ago I found a HN post doing exactly that -- and I finally took the time to set up use pagecrypt.

I tag private posts in the front matter with tags: ["private", "bla", bleh]. The compiled .html file results in an anchor element <a href="/tags/ private"> private </a>. I grep for this and password-protect it with npx pagecrypt.

I have a passwords.txt file that lets me set a different password for each file, with a default password as fallback:

default: wouldnt_you_like_to_know
_site/test_private_post/index.html: hunter2

I add the following action after npx eleventy:

password_file="passwords.txt"
default_password=$(grep "^default: " $password_file | cut -d ' ' -f 2)
for filename in $(grep -rl "/tags/ private" _site); do
    password=$(grep "^$filename: " $password_file | cut -d ' ' -f 2)
    if [ -n "$password" ]; then
        npx pagecrypt $filename $filename $password --iterations 2e6
    else
        npx pagecrypt $filename $filename $default_password --iterations 2e6
    fi
done

Check it out -- a super private post. The password is definitely not hunter2.

This is kind of a hack because I'm relying on the final .html file output having the text string /tags/ private in it. Additionally, any post that mentions /tags/ private will be password protected. So I've made sure to put a space here. This will change if I change the layout of my Nunjucks post template in the future. I'll keep looking for more elegant solutions.

Spring cleaning cruft

This website has been up at the same website since 2016 -- that's seven years now. I'm glad to have maintained continuity of the site, but there's a lot of cruft accumulated over the years. e.g. lots of orphan tags, weird bash scripts, commented out code, code in eleventy.js that I'm not sure does anything.

I moved/deleted/cleaned up various files.

Updating old descriptions

I cleaned up some of the old descriptions, updating the About and Resume pages.

What's next?

Not sure. I think there's more cruft I can slowly clean up. Does the website design look dated? I want to implement full text search eventually. Minisearch, Lunr and Fuse.js are good candidates.

How to draw isometric cubes with Javascript and HTML canvas

2023-03-15T00:00:00Z

How do you draw isometric tiles on a HTML canvas element?

A great first resource is Isometric graphics in Inkscape from Nicolás Guarín-Zapata's blog, but that's not going to work as-is for canvas. The difference is that in Inkscape you're drawing the square on the canvas before you make the transformations, but in HTML Canvas you're transforming the canvas first before you draw the square.

There are several key modifications you need to make:

Do the reverse of what's mentioned in the blog post: rotate, transform, then scale, in that order. This is because the complete transformation matrix is given by
$M_{\textrm{complete}} = M_{\textrm{rotation}} M_{\textrm{skew}} M_{\textrm{scaling}}.$
You need to translate the canvas such that it's centered on your square before rotating and skewing. In HTML canvas the CanvasRenderingContext2D.rotate() function the center of rotation is the top-left corner of the canvas and not a location relative to any shape.
You will have to play around with the positions and sizes of the faces if you want them to line up properly. I don't really understand exactly how In addition, note the skew transform for angle that are not 30 degrees. The transform should be Math.tan(angle * (90-this.angle*2) / this.angle * -1), not simply Math.tan(angle * -1). 30 degrees is a special case. This is useful for e.g. playing nicely with isometric pixel art assets that usually use a 26.5 degree angle.

Source code for the example above:

<canvas id="game" width="300" height="500"></canvas>
<script>
  let Game = document.getElementById("game");
  Game.ctx = Game.getContext("2d");
  class IsoRect {
    constructor(settings) {
      this.position = settings.position;
      this.width = settings.width;
      this.height = settings.height;
      this.direction = settings.direction;
      this.color = settings.color;
      this.angle = settings.angle ? settings.angle : 30
    }
    render() {
      Game.ctx.save();
      Game.ctx.fillStyle = this.color;
      const angle = (Math.PI / 180) * this.angle;
      Game.ctx.translate(
        this.position[0] + this.width / 2,
        this.position[1] + this.height / 2,
      );
      if (this.direction === "side") {
        Game.ctx.rotate(angle);
        Game.ctx.transform(1, 0, Math.tan(angle), 1, 0, 0);
        Game.ctx.scale(1, Math.cos(angle));
      } else if (this.direction === "front") {
        // front
        Game.ctx.rotate(-angle);
        Game.ctx.transform(1, 0, Math.tan(angle * -1), 1, 0, 0);
        Game.ctx.scale(1, Math.cos(angle));
      } else {
        // plant
        Game.ctx.rotate(angle);
        Game.ctx.transform(1, 0, Math.tan(angle * (90-this.angle*2) / this.angle *  -1), 1, 0, 0);
        Game.ctx.scale(1, Math.cos(angle));
      }
      Game.ctx.translate(
        -(this.position[0] + this.width / 2),
        -(this.position[1] + this.height / 2),
      );
      Game.ctx.fillRect(
        this.position[0],
        this.position[1],
        this.width,
        this.height,
      );
      Game.ctx.restore();
    }
  }

  new IsoRect({
    position: [100, 100],
    width: 100,
    height: 100,
    direction: "side",
    color: "red",
    angle: 26.5,
  }).render();
  new IsoRect({
    position: [188, 100],
    width: 100,
    height: 100,
    direction: "front",
    color: "tomato",
    angle: 26.5,
  }).render();
  new IsoRect({
    position: [144, 33],
    width: 100,
    height: 90,
    direction: "plant",
    color: "orange",
    angle: 26.5,
  }).render();


  new IsoRect({
    position: [100, 305],
    width: 100,
    height: 100,
    direction: "side",
    color: "red",
  }).render();
  new IsoRect({
    position: [186, 305],
    width: 100,
    height: 100,
    direction: "front",
    color: "tomato",
  }).render();
  new IsoRect({
    position: [143, 230],
    width: 100,
    height: 100,
    direction: "plant",
    color: "orange",
  }).render();
</script>

Can NFTs be used to fund art?

2023-01-01T00:00:00Z

Imagine you're an artist who's just created a piece of art--say, a novel. You'd like this piece of art to be enjoyed by people around the world, but at the same time, you need money to eat. The way you'd normally square this circle is to publish the novel and gate access to it, e.g. sell the eBook for $10, the paperback for $20, and only the people who have bought the book can read it.

That's the normal approach. Can we do something different? Here's a crazy thought: distribute the work completely free. Let anyone read it, make copies of it, send it to their friends, whatever. Let the work spread as widely as possible. Then, create a golden plaque signed by you, the artist (or maybe a signed copy of a gilded hardback version, whatever strikes your fancy). Then auction it off to the highest bidder.

The question is: would this work? Would this make the artist any money? Better yet, could an artist make more money under this post-hoc auction model compared to the regular model?

The auction approach seems quite compelling. As an artist, I'd want my work to be accessible to as many people as possible. I wouldn't want to deny others the pleasure of experiencing my work by gating it off -- especially when it costs me nothing to give someone a copy of an eBook or a JPEG. At the same time, though, I'd be really worried that the auction model wouldn't work; that nobody would pay me fairly for my work, that I wouldn't recoup the costs associated with the production of the work. So I'd need to be seriously convinced that I could make as much -- if not more -- money under the auction model.

I worked with Bassel to create a toy economic model of how this could work.

Protocols for optimal health

2023-01-01T00:00:00Z

Sleep

Entrain circadian clock with as much light as possible in the day, and avoid light as much as possible at night

Light entrainment happens solely in the upper visual field -- extraocular light perception does not exist.

Get light in the upper visual field.

Exercise

To be maximally healthy, we should have long duration steady state cardio, short and intense maximal heart rate cardio, and resistance training.

180 minutes of Zone 2 cardio
30 minutes of maximal HR cardio (certainly not sustained)
Resistance training

The primary driver of muscle hypertrophy is mechanical tension. Your muscles have to produce force to overcome a weight. By Newton's Third Law, a force likewise acts on your muscles. The cells in your body ha

"mechanotransduction"

Burkholder (2007). Mechanotransduction in skeletal muscle

There's some evidence that loading the muscle (ergo forcing it to produce force) in a stretched position (e.g. when the muscle is at its longest)

Nutrition

Optimising my onebag

2022-12-01T00:00:00Z

Things to remove

Timberland boots: Too heavy, can be replaced with something a bit more minimalistic, even for Iceland / really cold weather.
Uniqlo cargo pants: Too heavy of a weight, not necessary, and replaced with the water resistant pants anyway.
- In future, wear 1 pair of pants, pack 1 pair of pants, so I'll just go with grey slacks plus gray water resistant pants
Toiletries: Carrying way too much soap/shampoo, I'll just use one bar of soap for everything in the future.
IMDA green bomber jacket: Actually quite decent, dresses up nicely, but doesn't have much warmth at all, I feel like almost always i'd rather wear the UL down or an outer layer rather than this.

Things to add

The weird rubber thingys that let you strap stuff to the outside of the onebag that I saw in a Reddit post somewhere
Sunscreen
Button-up shirt: Get something nice enough to wear to formal dinners, but also casual enough to wear on its own, probably some sort of Merino wool shirt
- something long sleeve, light, breezy, can't be linen because that doesnt look formal enough? and it cannot ever wrinkle
- does such a thing even exist???
Sandals/slippers: Something open-toed, but with straps, used for lounging round the beach, walking around indoors, minimalist hiking/walking.
Phone case/stand of some sort: To prop up phone so I can have a more comfortable viewing angle when working on the go.
Bluetooth mouse. Not strictly necessary but it's a bit annoying to have to tap the screen of my phone to click stuff. Also useful for laptop obviously.
Cap/hat/visor (for hot weather): Something to keep the sun out of my eyes, good for walking on sunny days, Frisbee, etc.
Gloves: Not super pressing this time of year, but look out for a good, lightweight pair that works with touchscreens.
- I DEFINITELY needed gloves for skiing, and those gloves are bulky AF -- don't think they can fit in a onebag. Maybe Arcteryx or Patagonia sells some UL versions.
Merino buff: I see this recommended all the time on r/onebag. Neck and head protection for the cold, and even potentially the heat (avoid sunburn), impromptu sleep mask, face mask.
Sarong. This was such a good thing to wear around hostels, lounging round the house, even as an impromptu picnic mat, sunshade, etc. Thee batik pattern certainly attracted attention. Cotton isn't such a good material though, takes forever to dry (to be fair, it doesn't really have to be washed that often), maybe look for IDK, linen??

Things to replace:

Laptop: Biggest win in my opinion. Really full-featured, but way too big and heavy, and battery life is now poor. But running a proper desktop OS is non-negotiable, and Windows is an additional bonus for gaming (but I do very little of this while traveling). I'd like to replace it with something that has a much smaller form factor.
- Ideal device is something like a tablet form factor that's
  - runs Linux
  - folds into two
  - can also be used as a portable external monitor for the phone
- Maybe some sort of foldable portable monitor with an all in one "PC Stick", or Raspi with custom battery case, or the Framework Laptop case, idk
Phone: It's functional, but if it were foldable and ran Linux I wouldn't need a phone at all. I've recently been able to be more productive on the go by buying a foldable Bluetooth keyboard: small enough to pack in a coat pocket
- One incredibly underrated thing about a bluetooth keyboard is that apart from allowing you to type with all ten fingers, it also effectively doubles the phone's screen real estate (since when you type on your phone half of the screen is taken up by the on-screen keyboard)
Remarkable: Don't do enough reading on it, because I consume a lot of content online, and the friction of sending PDFs/eBooks to my RM is too great.
- Can be changed with the right habits. Keeping for now.
Rain shell. The existing outer layer I have sucks, not waterproof at all. Thinking of getting a Arcteryx shell of some sort as an outer layer that's waterproof but also breathable with pit zips etc, using it as the only layer on warmer days and adding in Uniqlo UL Down jacket + Heattech layers in cold weather.
Heattech thermal layers. Do I really need 2 pairs of each?
- The thinner blue layer is quite nice for going to the gym on slightly colder days, packs small anyway, so i'm fine
- I think the thing that takes the most space is the grey Heattech extra warm sweater layer, that one is indeed very warm, it's like a sweater of some sort almost, maybe I take it as double duty.

Maybe?

Some sort of sling or hot weather vest: backpacks go on the back, hard to slip stuff in and out quickly, I have very much appreciated the deep inner pockets you get on the Uniqlo UL Down jacket.
Relatedly, hot weather friendly outer shell -- UV protection, rain protection, with inner pockets and stuff, must be super breathable (pit zips are a must) maybe in a light grey colour, how do the Arcteryx stuff stack up in hot weather?

Things I want but rejected:

Cleats: Honestly this would be really nice to have to play sports but sacrifices have to be made while onebagging I guess.
- Why don't they make something like detachable cleats? LOL
Beanie: Really nice to have in cold weather, but they tend to stink because of my head oils. Find another solution.
Magic TCG deck. Lol

Other thoughts

Is there an outer shell that is suitable for wearing in hot weather?? Honestly having an outer shell that's waterproof in Singapore seems like a good shout.. but full black is going to be incredibly hot, maybe some sort of grey would be better.
Have been wearing the Vivobarefoot Primus Trail FG for two weeks now. Reasonably comfortable, doesn't smell (yet), have been putting it through its paces with frisbee and stuff. A bit less traction than I'd like on the hard muddy grass, I've slipped a couple of times.
Haven't found any use for the big metal container that I bought in Iceland, it is really nice that it's metal so you can heat it up, store hot food in it etc, but it just takes up so much bulk. I suppose I can use it as a container to store like idk, socks or underwear or something?
What do I do with the metal crampons that I bought, they are so heavy and useful for only one specific thing (hiking in Iceland in winter/spring). This is more general question that is worth considering: what do you do with all the one-use stuff you need for a specific usecase, can't keep bringing that stuff with you
The fanny pack is great, keeps all my stuff with me safe and sound, no complaints there. Good buy.

My "indefinite" onebag setup

2022-12-01T00:00:00Z

Der Schatz von Hiddensee

2022-10-01T00:00:00Z

`````````````````````

Der Schatz von Hiddensee

Winterferien

"Annika, bist du fertig? Wir warten auf dich!"

Familie Sonntag will in Urlaub fahren. Die Koffer sind schon im Auto, aber Annika ist noch nicht da.

"Ja, ich komme gleich. Ich shreibe nur noch schnell eine SMS an Nora!", ruft Annika.

"Das kannst du auch im Auto machen! Komm, wir wollen fahren. Deine Schwester wartet schon auf uns."

Nora ist Annikas ältere Schwester. Sie ist Studentin in Greifswald. Alle haben Ferien und wollen zusammen auf der Insel Hiddensee Urlab machen. Familie Sonntag wohnt in Dresden. Sie fahren mit dem Auto über die Autobahn in Richtung Berlin und dann weiter nach Greifswald. Da holen sie Nora ab und fahren dann zusammen weiter an die Ostsee.

"Mama, wie lange fahren wir denn bis Greifswald?" will Annika wissen, als sie im Auto sitzen und losfahren.

"Ungefähr vier bis fünf Studen", antwortet Frau Sonntag.

"Und wie lange fahren wir dann von Greifswald bis Hiddensee?" fragt Annika.

"Das dauert noch einmal ungefähr zwei oder drei Stunden", sagt ihr Vater.

Sie fahren durch das Zentrum von Dresden und kommen an der Frauenkirche vorbei. Die Kirche var viele Jahre eine Ruine, aber seit ein paar Jahren ist sie fertig restauriert. Es gibt viele Touristen, die die Kirche sehen wollen. Danach kommen sie an der Semperoper vorbei.

"In die Semperoper müssen wir bald wieder gehen. Wir waren schon lange nicht mehr im Konzert", sagt Herr Sonntag.

"Stimtt, das letzte Konzert war ein Mozart-Konzert. Das war so schön!" erinnert sich Frau Sonntag. "Vielleicht gibt es ja auf Hiddensee ein klassiches Konzert. Dann können wir da alle yusammen hingehen!"

Aber Annika interessiert sich nicht für klassische Musik oder Opern. Sie hört lieber Popmusik. Im Sommer gibt es in Touristenorten auch Konzerte für junge Leute, aber im Winter? Da ist es sicher sehr langweillig auf der Insel. Annika freut sich nicht auf diese Ferien. Ihre Klassenkameraden fliegen auf die Kanarischen Inseln oder nach Florida oder gehen in den Alpern Skifahren. Aber ihre Familie macht Urlaub in Deutschland - auf einer Mini-Insel. Wie langweillig. Gut, dass Nora mitkommt! Vielleicht hat sie ein paar coole Ideen, was man auf der Insel so machen kann. Weil die Fahrt ein paar Stunden dauert, schaltet sie ihren MP3-Player ein. Bis sie Nora treffen, kann sie noch viele Songs hören.

Unterwegs zur Insel

Farming snails in Singapore

2022-08-01T00:00:00Z

Bigger picture is Alternative Agriculture

Alternative agriculture

2022-08-01T00:00:00Z

Snail farming

Also called heliciculture. Farming snails in Singapore

Should we bring back the triclinium?

2022-05-01T00:00:00Z

https://blogs.getty.edu/iris/reclining-and-dining-and-drinking-in-ancient-rome/

https://books.google.com.sg/books/about/The_Roman_banquet.html?id=PaAk2tWYTEQC&redir_esc=y

https://www.google.com/search?q=the+roman+banquet&hl=en&source=lnms&tbm=isch&sa=X&ved=2ahUKEwjZyJS724LxAhXUmuYKHXZkDKQQ_AUoAXoECAcQAw&biw=1715&bih=927&dpr=1#imgrc=x26EGEAecc9zpM

https://www.npr.org/sections/thesalt/2019/05/20/712772285/the-lavish-roman-banquet-a-calculated-display-of-debauchery-and-power

https://www.througheternity.com/en/blog/things-to-do/banquets-ancient-rome.html

https://en.wikipedia.org/wiki/Food_and_dining_in_the_Roman_Empire#/media/File:Triclinium_-_Arch%C3%A4ologische_Staatssammlung_M%C3%BCnchen.JPG

Ants

2022-04-23T00:00:00Z

Toh gave this puzzle to me and Wilfred in Gold Mine; coded this on Saturday afternoon with Rayhan before going to Croatia

One hundred ants are dropped on a meter stick. Each ant is traveling either to the left or the right with constant speed 1 meter per minute. When two ants meet, they bounce off each other and reverse direction. When an ant reaches an end of the stick, it falls off. At some point all the ants will have fallen off. The time at which this happens will depend on the initial configuration of the ants.

Question

Over ALL possible initial configurations, what is the longest amount of time that you would need to wait to guarantee that the stick has no more ants?

Answer

One minute, regardless of the number of ants.

Proof by animation

Bibliography

Recontextualising my site (2022)

2022-02-03T00:00:00Z

Introduction

The last time I redesigned my website was 2020. I moved from Jekyll to Eleventy and made the site look a bit nicer. Now I want to do a bit more with it: not so much a redesign, but rather a recontextualisation. What do I want my site to be and do?

Full-text search

I want people to be able to discover my website with a Ctrl-K input, something like Notion, or the full text search in Visual Studio Code Ctrl-Shift-F. Ideally they should be able to discover all the tags as well.

Joel Lee, [3/2/2022 1:53 PM] > [In reply to lieu] > https://docsearch.algolia.com/ ?

lieu, [3/2/2022 1:55 PM] > [In reply to Joel Lee] Yeah i know algolia but it seems a bit heavy and or overkill

https://lunrjs.com/

This is the kind of search I'm looking for:

https://braindump.jethro.dev/search/?query=food

https://github.com/jethrokuan/braindump I think it's built into his template so he didn't have to implement

https://thesephist.com/posts/monocle/

I really like Monocle, this is almost exactly what I want

Password-protect private posts

Right now, private posts are on the website, but they're just not displayed publicly (you have to have the exact URL). This is OK for now but will obviously become an issue once full-text search is implemented.

I would ideally like to index private posts as well (make them searchable) but password-protect the full text content.

Can you gain muscle doing only isometric exercises?

2022-02-02T00:00:00Z

Hypothesis

Oranchuk et al. (2018) performed a meta-analysis of twenty-six studies and found that "substantial improvements in muscular hypertrophy and maximal force production were reported regardless of training intensity".

maximum voluntary isometric contraction (MVIC)

Unfortunately "most studies (16/26) recruited untrained participants, while the remainder (11/26) utilized "active" or "recreationally trained" participants", so the evidence isn't strong whether isometric training can result in hypertrophy in well-trained individuals.

Skeletal muscle hypertrophy in response to isometric, lengthening, and shortening training bouts of equivalent duration

Practical applications

Is nonsurgical penile enhancement possible?

2022-01-25T00:00:00Z

Key questions

Are there nonsurgical methods that (semi-)permanently increase flaccid length?
Are there nonsurgical methods that (semi-)permanently increase erect length?
Are there nonsurgical methods that (semi-)permanently increase penile girth?
If any of 1 to 3 hold, how do these methods work mechanistically/what is their anatomical basis?
What are the potential gains and risks of these methods?

Motivation

Introduction

Good study by Moncada et al. (2022). RCT. n=93 patients with Peyronie's (curved penis): penile traction devices are effective at decreasing curvature and increasing stretched penile length: "SPL increased significantly in the PG compared to baseline and compared with the NIG, ranging from 0.5 to 3.0 cm (mean 1.8 cm; P < 0.05)"

Not so good study: Nikoobakht et al. (2011). n=23 patients who complained of "short penis" were given their "Golden Erect" (lmao) penile traction device. Mean stretched penile length also significantly increased from 11.5 ± 1.0 cm to, respectively, 12.4 ± 1.3 cm and 13.2 ± 1.4 cm during the first and second follow-up (P < 0.05).

Mechanism of penile growth

Initial length gains come from loosening of the suspensory ligament; further gains come from tunica stretching.

Reddit post

Length gains are caused by a few things.

Erection quality improvements, loosening the suspensory ligaments and stretching of the tunica and increase in blood vessel volume.

Erection Quality is perhaps the easiest and quickest way to gain. Simply put, erection quality is determined by much blood a penis holds during an erection. The more blood it can hold, the stronger the erection.

Gains from EQ typically are about .25 of an inch for a man with okay EQ. Bad EQ can be as much as .5 of an inch

EQ improvements happen in the first 4 weeks of PE. Most PE techniques improve EQ passively over time but for active develop, I recommend using an angion method for 2-3 weeks. (A post on angion method is TBD)

The suspensory ligaments connect at the base of the pubic bone. They are what attach the penis to the body. They play a role in erection angle as well as length. Loosening the Suspensory ligaments allow more of the chambers of the penis to stick out of the body, as the chambers start much farther back into pelvic region(3-6 inches)

Gains from loosening of the suspensory ligaments usually happens within two months. Depending on how tight they were at the beginning the average man can expect to gain .25-.5 of an inch Targeting the suspensory ligaments.

Stretching the tunica takes much more time compared to EQ gains or suspensory gains. The tunica surrounds the shaft like a wrapper. This is what gives the penis it's hardness during an erection, otherwise it would be very spongy, like the glans/head. What causes growth in this fashion is the healing of micro damage from stretching the penis in multiple ways. Daily length sessions cause the microtears generated to heal in an elongated state leading to a longer shaft.

Blood vessel expansion will be covered more with girth

Gains from tunica stretching are about .5 of inch every 6 months given commitment and correct use of progress overload.

Given all three of the of the ways to a gain a man new to PE can gain 1 inch in as little as 6 months but typically it is about 4 -14 months, depending on their bodies response to the stimulus, their starting size, where larger men tend to grow faster, proper techniques, and personal commitment.

evidence for suspensory ligament loosening: see Protogerou et al. (2019).

Still-unanswered questions

How does tension actually induce growth? Is it actually microtears? It's possible. We know that isn't the mechanism in muscular hypertrophy. but the tunica is fibrous tissue that is made up of elastin and collagen. But is it microtearing that causes the lengthening or some other mechanism causing the plastic deformation?

Bibliography

Most helpful article so far: Protogerou V, Anagnostopolou S, Venierates D, Troupis T, Plagou M, Vlassis K, Skandalakis P. Penis ligaments: their use in "increasing" the size of the penis in penile augmentation procedures. Anatomical description in human cadavers and clinical results of a phalloplasty series. Ann Ital Chir. 2010 May-Jun;81(3):199-204. PMID: 21090558.

Zhang X, Huang Z, Xiao Y, Kuang L, Zhang M, Zhang G, Li Q, Bai W, Xu T. Suspensory ligament release combined with acellular dermal matrix filler in infrapubic space: A new method for penile length augmentation. Andrologia. 2019 Oct;51(9):e13351. doi: 10.1111/and.13351. Epub 2019 Jul 2. PMID: 31264245. https://pubmed.ncbi.nlm.nih.gov/31264245/

Moncada I, Krishnappa P, Romero J, Torremade J, Fraile A, Martinez-Salamanca JI, Porst H, Levine L. Penile traction therapy with the new device 'Penimaster PRO' is effective and safe in the stable phase of Peyronie's disease: a controlled multicentre study. BJU Int. 2019 Apr;123(4):694-702. doi: 10.1111/bju.14602. Epub 2018 Nov 22. PMID: 30365247.

Is sauna good for you?

2022-01-23T00:00:00Z

Key questions

Is sauna good for you?
In what ways is it good for you?
Why is sauna good for you? What sort of mechanistic pathway does it act over?
What are health risks of sauna?

Bibliography

Laukkanen JA, Laukkanen T, Kunutsor SK. Cardiovascular and Other Health Benefits of Sauna Bathing: A Review of the Evidence. Mayo Clin Proc. 2018 Aug;93(8):1111-1121. doi: 10.1016/j.mayocp.2018.04.008. PMID: 30077204. Laukkanen JA, Kunutsor SK. Is sauna bathing protective of sudden cardiac death? A review of the evidence. Prog Cardiovasc Dis. 2019 May-Jun;62(3):288-293. doi: 10.1016/j.pcad.2019.05.001. Epub 2019 May 16. PMID: 31102597.

How to import and display Markdown files in React

2021-09-01T00:00:00Z

Recently I found myself in need of importing and displaying long documents in a React SPA, but I didn't know how to do it. I combined the answers in these StackOverflow questions here ( 1, 2).

Three simple steps:

First, install React Markdown with yarn add react-markdown or your preferred package manager.

Then, declare a markdown.d.ts file and put it in your folder (I believe anywhere that Typescript can see is OK):

declare module '*.md' {
	const value: string; // markdown is just a string
	export default value;
}

You can then import Markdown files, fetch them, and parse them into a string. Pass the parsed Markdown file into the ReactMarkdown component, which takes in a children: string prop. You can use the useState and useEffect hooks to do this cleanly:

import { useState, useEffect } from 'react'
import ReactMarkdown from 'react-markdown'

// Terms of Use and Privacy Policy
import TermsOfUse from './texts/terms_of_use.md'
import PrivacyPolicy from './texts/privacy_policy.md'

export const TermsPage = (): JSX.Element => {
	const [tosText, setTosText] = useState('')
	const [privacyPolicyText, setPrivacyPolicyText] = useState('')

	// Fetch Terms of Use
	useEffect(() => {
		fetch(TermsOfUse).then(res => res.text()).then(text => setTosText(text))
	})

	useEffect(() => {
		fetch(PrivacyPolicy).then(res => res.text()).then(text => setPrivacyPolicyText(text))
	})

	return (
		<div>
			<ReactMarkdown children={tosText} />
			<ReactMarkdown children={privacyPolicyText} />
		</div>
	)
}

This means you could probably use ReactMarkdown + Router for your blog instead of a static site generator. Maintain a map of routes to Markdown files and fetch the file when the user navigates to the chosen route. Maybe (most probably) I've just reinvented the wheel, lol.

How to implement role-based access control for AWS (Cognito, DynamoDB)

2021-06-22T00:00:00Z

Introduction

We have projects. Each project can have one or more S3 buckets. Each project can have one or more users. Each user can belong to one or more projects. (There is a one-to-many mapping between projects and buckets, and a many-to-many mapping between users and projects).

We want to make sure that a user can log in and access only the buckets that are part of projects he belongs to. As Amazon AWS is a huge and very complicated system, there were many ways to do it, all of which seemed plausible. I worked with Chris to explore different approaches, eventually settling on the current one. In this design document I list the five different approaches, explore their advantages and disadvantages, and justify why I chose the implementation we did.

Approaches

Five different approaches were tried and four were rejected. I eventually settled on using the AssumeRole method which I believe is the correct tradeoff.

Using IAM User/Role/Bucket policies [Rejected]
Using Cognito User Groups [Rejected]
Using Attribute-based access control [Rejected]
Using AssumeRoleWithWebIdentity [Rejected]
Using AssumeRole [Accepted]

Using IAM User/IAM Role/Bucket policies [Rejected]

When a user is onboarded, we create an IAM User and a Cognito User in the Cognito User Pool.

When the person logs into the Cognito user pool, the identity pool will give him this IAM Role using the JWT loop with the same policies as the Bedrock/Talend IAM user.

On each bucket we create a policy that only allows a particular IAM Role/IAM user to access it.

The user flow is as follows:

Use Session Token to assume IAM Role.
Use IAM Role to access S3 Bucket.

The dealbreaker of this approach is that every user would potentially have to have a unique IAM User/Role, and we couldn't attach all of them to a bucket. This is not scalable as there is a maximum policy length of the bucket, and as you add more users, eventually the bucket will not be able to accommodate more users. We need a way to do group-based authentication, which is what I turned to next.

Using Cognito User Groups [Rejected]

The disadvantage of the previous method is that there was no grouping.

So now we institute a one-to-one-mapping between projects and Cognito user groups. Each project will be a Cognito user group, and that user group would allow access to all buckets in that user group.

The onboarding flow is as follows:

Create bucket. Get bucket ARN.
[Requires bucket ARN] Create IAM role that allows access to bucket. Get IAM role.
[Requires IAM Role ARN] Create Cognito User Group, passing in IAM Role ARN.
[Requires Cognito User Group ARN] Set bucket policy on S3 bucket allowing user group access.

Putting the permissions on the user group is done as follows:

{
  // ...
  Effect: "Allow", 
  Resource: [
    `arn:aws:s3:::${bucketName}`,
    `arn:aws:s3:::${bucketName}/*`,
  ]
  }

Users would get their IAM Role via getCredentialsForIdentity.

The main disadvantage of this approach is that many front end requests would be required to list all groups a user belonged to. When a user logs in, he has to assume a role. But one role is mapped one-to-one to a project. So what happens if a user is part of multiple projects? Then he has to get the first user group role, list the buckets in that project, then switch to the second user group role, list the buckets in that project, and so on. Chris and I felt that this was untenable.

You could try to have a "super user group" that encompassed the permissions of multiple IAM roles. But this could conceivably lead to an exponential number of user groups. For example, if you have 3 projects (A, B, C), then you would need 7 user groups to cover all eventualities: A, B, C, AB, AC, BC, ABC. And if you have 4 projects then you need 15 user groups, and so on... We decided that this approach was not feasible.

Attribute-based access control [Rejected]

The next idea was to put attributes on Cognito users, where each attribute is a project. For example, if the user has three different projects, then he would have {project1: true, project2: true, project16: true}. These project names would have to be globally unique. Then you would create a global policy that grants permissions to the project's resources as long as the user has that project name in his tags.

This approach was recommended by the AWS ABAC tutorial:

You can map the membership attribute to a tag key for principal to be passed on to the IAM permissions policy. This way you can create a single permissions policy and conditionally allow access to premium content based on the value of membership level and tag on the content files.

This approach had several advantages. The first advantage is that you only need one front-end request, and you don't need an exponential number of user groups. It would allow fine-grained user permissions: for example, you could have {project1dev: true, project2dev: true, project16admin: true}, to make the user an admin of project 16.

The key disadvantages was that for this method to work, we would have to have one long policy document that grants permissions to the project's resources as long as the user has that project name in his tags. And every time a new project was created, we would have to append to the policy document. But there is a maximum character limit on a single permissions policy, which would impose a limit on the total number of projects in our system.

While there's a way to get around this with a slightly different ABAC mechanism, there is no free lunch. We faced a dilemma: either we limit the total number of projects per user, or you limit the total number of projects in our system. Eventually we decided to try something else.

AssumeRoleWithWebIdentity [Rejected]

Idea here is to dynamically generate access control based on the WebIdentity provided. We look up the user in the DynamoDB table, check what projects the user belongs to, and builds a custom IAM Role that allows the user the ability to access only those buckets.

User flow:

User signs in with Cognito User Pool to get a JWT.
User calls getId to get an identity pool ID.
User calls AssumeRoleWithWebIdentity with the identity pool ID.
The Lambda function looks at the ID, looks up the DynamoDB table to check the projects the user is in, and builds a custom IAM role which is the intersection of the base IAM role (allows access to all S3 buckets) and the dynamically-generated InlinePolicy.

Advantages:

The key advantage is that this allows us to dynamically generate a custom IAM role depending on the user's projects, which solves a lot of the problems of the previous approaches:

It would require fewer requests to switch roles and access buckets from different projects, making the frontend development much simpler. In the role-based control approach a user has multiple roles, and the frontend needs to make multiple getCredentialsForidentity calls to switch between these roles in order to access buckets from different projects. The new ABAC system would not require switching roles with getCredentialsForIdentity as there is only one role for that particular identity.
It would potentially allow more fine-grained control of project resources. For example, a user group might have "superusers" and "regular users", and this could be reflected in the DynamoDB. In fact, you can put any type of condition/property in the DynamoDB and we could check for that.

The dealbreaker disadvantage is that this approach is not secure, because AssumeRoleWithWebIdentity is a call that does not require credentials to make. (This is not quite the same as an unauthenticated endpoint: you still need a valid Cognito OpenID token). Why is this a problem? Because this means a user could technically write their own custom frontend and call AssumeRoleWithWebIdentity themselves. They could then pass in an empty inline policy, which would give them access to all S3 buckets. While this is a bit of an edge case, we thought this was nevertheless a security flaw and abandoned this idea.

AssumeRole [Accepted]

Like before, the idea here is to dynamically generate the IAM role based on the user.

User flow:

User signs in with Cognito User Pool to get a JWT Bearer Token.
User calls the auths/resources endpoint using a GET request, passing in the JWT Bearer Token.
The auths/resources endpoint calls a Lambda function, which looks at the user ID contained in the JWT Bearer Token, looks up the DynamoDB table to check the projects the user is in.
The Lambda function generates a custom InlinePolicy from the information it obtains in the DynamoDB Table.
The lambda function calls sts.assumeRole to generate a custom IAM role, which is the intersection of the base IAM role (allows access to all S3 buckets) and the dynamically-generated InlinePolicy.

The auths/resources endpoint is private. Anyone who tries to access it without a valid JWT Bearer token will be given a 401 Forbidden error.

This approach is very similar to the previous AssumeRoleWithWebIdentity. The key difference is that the user himself cannot call assumeRole. I have given permissions only for the Lambda function to call assumeRole. So the user cannot create his own frontend like he could in AssumeRoleWithWebIdentity.

Advantages:

Secure
No exponential number of user groups
No need to switch roles
Fine-grained access control

Disadvantages:

There is a maximum character limit in a single permissions policy (10240 chars). This means that we can only have at most ~50 concurrent projects per user. We decided that this was an acceptable limit.

AssumeRole's Base IAM Role

The Base IAM role allows access to all S3 buckets, and looks like this:

AllowAccessToAllS3BucketsRole:
  Type: "AWS::IAM::Role"
  Properties:
    RoleName: ${self:custom.IAMRolePrefix}-allow-access-to-all-s3-buckets-role
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Effect: 'Allow'
          Principal:
            AWS:
              Fn::GetAtt: 
                - AllowAssumeRoleRole
                - Arn
          Action:
            - sts:AssumeRole
    PermissionsBoundary: 'arn:aws:iam::052567997892:policy/GCCIAccountBoundary'
    Policies:
      - PolicyName: 'allow-access-to-all-s3-buckets'
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            Action:
              - s3:GetObject
              - s3:PutObject
              - s3:PutObjectAcl
              - s3:PutObjectVersionAcl
              - s3:ListBucket
            Effect: Allow
            Resource:
              - arn:aws:s3:::*
              - arn:aws:s3:::*/*
            Condition:
              Bool:
                aws:SecureTransport: true

For security purposes, only the AllowAssumeRoleRole is allowed to assume this role. This base role should not be passed to the user. Instead, we will create a new role like this:

export async function assumeRole() {
  const baseRoleArn = 'arn:aws:iam::052567997892:role/iamrole-imda-psf-lieudev-allow-access-to-all-s3-buckets-role'
  const inlinePolicy = ... // create the inline policy here after a DynamoDB query
  const assumeRoleParams = {
    RoleArn: baseRoleArn, // Role the caller is assuming (base role). In this case, AllowAccessToS3Buckets
    RoleSessionName: 'TestUser', // pass the name associated with the user
    Policy: inlinePolicy,
    DurationSeconds: 3600,
  }

  try {
    const sts = new AWS.STS()
    const tokenResult = await sts.assumeRole(assumeRoleParams).promise()
    ...

We can see that we pass in inlinePolicy into assumeRoleParams, which creates a new role which is the intersection of inlinePolicy and baseRoleArn. This will allow only access to resources that are allowed inside inlinePolicy.

Solving two brainteasers 舅母 gave me

2021-05-09T00:00:00Z

Introduction

I was given these problems by my 舅母:

The questions are actually quite poorly written (in the sense that they are ambiguous) but the ambiguity actually makes them inadvertently more challenging and thus more interesting.

Swimmers and non-swimmers

The native inhabitants of a certain island country have a strange characteristic: those who are swimmers always tell the truth. But those who are nonswimmers always lie. Suppose that you visit the island and meet a group of three people who are native inhabitants. You ask the first person, "Are you a swimmer?" The answer is unintelligible. The second person immediately says, "That was 'Yes.' I am a swimmer, too." The third person says, "They are both lying." Is each of these three people a swimmer or a nonswimmer?

Answer: It depends on how you interpret "nonswimmers always lie". Possible answers: SSN, NNS, SNN.

Firstly, let's note that even though the answer of the first person is unintelligible, it must always be "yes". This is because a swimmer will always honestly answer "Yes" and a nonswimmer will always lie and say "Yes" too.

The answer to this question depends completely on how you interpret the phrase "nonswimmers must always lie"! Depending on your interpretation, possible solutions to this problem are two swimmers and a nonswimmer (SSN), NNS, and even SNN.

Suppose I always lie. Could I have said the statement "My name is Zheng Hong. I am eight feet tall."? That depends on how you interpret "always lie". Does it mean that every sentence in the utterance must be false? If so, I could not have said that, since "My name is Zheng Hong" is true. But if you take the utterance as a whole, then I could have said that, since the utterance as a whole is a lie.

In the same way, the sentence is "That was 'Yes'. I am a swimmer, too.". If "must always lie" means "every sentence must be a lie", then a nonswimmer could not possibly have said that, because "That was 'Yes'" is true. Hence the second person must have been a swimmer. But if we consider two sentences as one combined utterance, then the nonswimmer could have said that, since the combined utterance is false, and this would still follow the rule of "nonswimmers always lie". Then the second person could be a nonswimmer.

Let's consider the first situation where the second person must have been a swimmer. Then the first person must have been a swimmer. This is because if the first person was a nonswimmer, then the second person would not have said "I am a swimmer, too". The word "too" implies that the first person was a swimmer, but this would be a false statement if the first person was a nonswimmer. Hence the first person must have been a swimmer. But if both first and second are swimmers, then the last person must be a nonswimmer, because "They are both lying" must be a lie, since neither of them are lying. Hence, SSN.

Now let's consider the situation where the second person is a nonswimmer (?N?). In which case, the first person could have been a swimmer or nonswimmer. Let's consider swimmer first. SN?. Is the third person a swimmer or a nonswimmer? Let's review:

"Yes" (true)
"That was 'Yes'. I am a swimmer, too" (false)
"They are both lying".

Sentence 3 must be a lie, since only one of them is lying. Hence, the third person is a nonswimmer. Thus, SNN.

We can also consider the case where the first person is a nonswimmer:

"Yes" (false)
"That was 'Yes'. I am a swimmer, too" (false)
"They are both lying".

Here Sentence 3 must be the truth, since both of them are lying. Hence, the third person is a swimmer. Thus, NNS.

In fact, we can run through all 8 possible combinations of swimmers and non-swimmers:

SSS: impossible, since the third person would not say that they are both lying.
NSS: impossible, since the second person would not say that he "too" was a swimmer
SNS: impossible, since the third person would say that only the second person is lying
SSN: possible
NNS: possible
NSN: impossible, since the second person would not say that he "too" was a swimmer
SNN: possible
NNN: impossible, since the sentence "They are both lying" would be true if so.

Custom Dice

What whole numbers should appear on the twelve faces of two number cubes so that the probabilities that the top two faces show any sum from 1 to 12 inclusive are exactly the same?

Answer: There are infinitely many whole number solutions.

First of all, let's note that we can fill all the faces with '0', or '13', '14', etc. Then the probability that the top two faces show any sum from 1 to 12 will be 0, which is exactly the same. So this is technically a solution. But of course this is a trivial solution. We should look for non-zero probabilities.

So the idea is as follows. Consider a "counter" that counts the number of ways each number can be rolled. As an example, the counter for two regular 6-sided dice would look like this:

{
    2: 1,
    3: 2,
    4: 4,
    5: 4,
    6: 5,
    7: 6,
    8: 5,
    9: 4,
    10: 4,
    11: 2,
    12: 1,
}

For example, there are six ways to roll a 7: (1,6), (6,1), (2,5), (5,2), (3,4), (4,3), and that's why we have the row 7: 6.

For this question, we want a counter that looks like this:

{ 0: whatever, we don't care 1: X 2: X 3: X ... 11: X 12: X 13: whatever, we don't care 14: whatever, we don't care ... ... 25: whatever, we don't care }

where all the X's are the same (and ideally not 0).

You can actually solve this problem by thinking about it, and filling the dice from the smallest number onwards. First you know that you need a 0, because you can only roll a sum of 1 if you roll a 0 and a 1. And given that there are only 36 (6x6) possible outcomes of the dice, you should figure that there can only be 1, 2, or 3 occurrences of 1--12. I guessed 2, and indeed I did it by hand. But in fact, there are infinitely many solutions! Here is some code to find solutions:

import itertools, collections

def is_equal_probability(c: collections.Counter, start: int, end: int):
    """
    Check if all numbers from start to end (inclusive) 
    have the same number of occurrences.
    """
    occurrences = c[start]
    for i in range(start, end+1):
        if c[i] != occurrences:
            return False 
    return True

# Generate 6-sided dice with the faces 0 to 12
# 13 choose 6: 1716 unique combinations
for dice1 in itertools.combinations(range(0, 13), 6): 
    for dice2 in itertools.combinations(range(0, 13), 6):
        # Get all the 36 dice rolls
        combinations = list(itertools.product(dice1, dice2))
        assert(len(combinations) == 36)
        # Once we have the 36 dice rolls, 
        # sum them up and count the number of each occurrence
        c = collections.Counter([sum(_) for _ in combinations])
        if c[1] > 0 and is_equal_probability(c, 1, 12):
            print("Found a pair of dice that have equal probability of summing to a value from 1 to 12!")
            print(f"Dice 1: {dice1}") 
            print(f"Dice 2: {dice2}")
            print(f"The 36 possible combinations: {combinations}")
            print(f"And their respective sums: {sorted(c.items())}")

And here's the output:

(base) lieu@DESKTOP-NV19JNO:~/dev/lieuzhenghong.com/root$ python3 dice.py 
Found a pair of dice that have equal probability of summing to a value from 1 to 12!
Dice 1: (0, 1, 2, 3, 4, 10)
Dice 2: (0, 1, 5, 6, 11, 12)
The 36 possible combinations: [(0, 0), (0, 1), (0, 5), (0, 6), (0, 11), (0, 12), (1, 0), (1, 1), (1, 5), (1, 6), (1, 11), (1, 12), (2, 0), (2, 1), (2, 5), (2, 6), (2, 11), (2, 12), (3, 0), (3, 1), (3, 5), (3, 6), (3, 11), (3, 12), (4, 0), (4, 1), (4, 5), (4, 6), (4, 11), (4, 12), (10, 0), (10, 1), (10, 5), (10, 6), (10, 11), (10, 12)]
And their respective sums: [(0, 1), (1, 2), (2, 2), (3, 2), (4, 2), (5, 2), (6, 2), (7, 2), (8, 2), (9, 2), (10, 2), (11, 2), (12, 2), (13, 2), (14, 2), (15, 3), (16, 2), (21, 1), (22, 1)]
Found a pair of dice that have equal probability of summing to a value from 1 to 12!
Dice 1: (0, 1, 2, 3, 9, 11)
Dice 2: (0, 1, 4, 5, 8, 12)
The 36 possible combinations: [(0, 0), (0, 1), (0, 4), (0, 5), (0, 8), (0, 12), (1, 0), (1, 1), (1, 4), (1, 5), (1, 8), (1, 12), (2, 0), (2, 1), (2, 4), (2, 5), (2, 8), (2, 12), (3, 0), (3, 1), (3, 4), (3, 5), (3, 8), (3, 12), (9, 0), (9, 1), (9, 4), (9, 5), (9, 8), (9, 12), (11, 0), (11, 1), (11, 4), (11, 5), (11, 8), (11, 12)]
And their respective sums: [(0, 1), (1, 2), (2, 2), (3, 2), (4, 2), (5, 2), (6, 2), (7, 2), (8, 2), (9, 2), (10, 2), (11, 2), (12, 2), (13, 2), (14, 2), (15, 2), (16, 1), (17, 1), (19, 1), (21, 1), (23, 1)]
Found a pair of dice that have equal probability of summing to a value from 1 to 12!
Dice 1: (0, 1, 2, 3, 9, 12)
Dice 2: (0, 1, 4, 5, 8, 11)
The 36 possible combinations: [(0, 0), (0, 1), (0, 4), (0, 5), (0, 8), (0, 11), (1, 0), (1, 1), (1, 4), (1, 5), (1, 8), (1, 11), (2, 0), (2, 1), (2, 4), (2, 5), (2, 8), (2, 11), (3, 0), (3, 1), (3, 4), (3, 5), (3, 8), (3, 11), (9, 0), (9, 1), (9, 4), (9, 5), (9, 8), (9, 11), (12, 0), (12, 1), (12, 4), (12, 5), (12, 8), (12, 11)]
And their respective sums: [(0, 1), (1, 2), (2, 2), (3, 2), (4, 2), (5, 2), (6, 2), (7, 2), (8, 2), (9, 2), (10, 2), (11, 2), (12, 2), (13, 3), (14, 2), (16, 1), (17, 2), (20, 2), (23, 1)]
Found a pair of dice that have equal probability of summing to a value from 1 to 12!
Dice 1: (0, 1, 2, 9, 10, 11)
Dice 2: (0, 1, 3, 4, 6, 7)
The 36 possible combinations: [(0, 0), (0, 1), (0, 3), (0, 4), (0, 6), (0, 7), (1, 0), (1, 1), (1, 3), (1, 4), (1, 6), (1, 7), (2, 0), (2, 1), (2, 3), (2, 4), (2, 6), (2, 7), (9, 0), (9, 1), (9, 3), (9, 4), (9, 6), (9, 7), (10, 0), (10, 1), (10, 3), (10, 4), (10, 6), (10, 7), (11, 0), (11, 1), (11, 3), (11, 4), (11, 6), (11, 7)]
And their respective sums: [(0, 1), (1, 2), (2, 2), (3, 2), (4, 2), (5, 2), (6, 2), (7, 2), (8, 2), (9, 2), (10, 2), (11, 2), (12, 2), (13, 2), (14, 2), (15, 2), (16, 2), (17, 2), (18, 1)]
Found a pair of dice that have equal probability of summing to a value from 1 to 12!
Dice 1: (0, 1, 3, 4, 6, 7)
Dice 2: (0, 1, 2, 9, 10, 11)
The 36 possible combinations: [(0, 0), (0, 1), (0, 2), (0, 9), (0, 10), (0, 11), (1, 0), (1, 1), (1, 2), (1, 9), (1, 10), (1, 11), (3, 0), (3, 1), (3, 2), (3, 9), (3, 10), (3, 11), (4, 0), (4, 1), (4, 2), (4, 9), (4, 10), (4, 11), (6, 0), (6, 1), (6, 2), (6, 9), (6, 10), (6, 11), (7, 0), (7, 1), (7, 2), (7, 9), (7, 10), (7, 11)]
And their respective sums: [(0, 1), (1, 2), (2, 2), (3, 2), (4, 2), (5, 2), (6, 2), (7, 2), (8, 2), (9, 2), (10, 2), (11, 2), (12, 2), (13, 2), (14, 2), (15, 2), (16, 2), (17, 2), (18, 1)]
Found a pair of dice that have equal probability of summing to a value from 1 to 12!
Dice 1: (0, 1, 4, 5, 8, 11)
Dice 2: (0, 1, 2, 3, 9, 12)
The 36 possible combinations: [(0, 0), (0, 1), (0, 2), (0, 3), (0, 9), (0, 12), (1, 0), (1, 1), (1, 2), (1, 3), (1, 9), (1, 12), (4, 0), (4, 1), (4, 2), (4, 3), (4, 9), (4, 12), (5, 0), (5, 1), (5, 2), (5, 3), (5, 9), (5, 12), (8, 0), (8, 1), (8, 2), (8, 3), (8, 9), (8, 12), (11, 0), (11, 1), (11, 2), (11, 3), (11, 9), (11, 12)]
And their respective sums: [(0, 1), (1, 2), (2, 2), (3, 2), (4, 2), (5, 2), (6, 2), (7, 2), (8, 2), (9, 2), (10, 2), (11, 2), (12, 2), (13, 3), (14, 2), (16, 1), (17, 2), (20, 2), (23, 1)]
Found a pair of dice that have equal probability of summing to a value from 1 to 12!
Dice 1: (0, 1, 4, 5, 8, 12)
Dice 2: (0, 1, 2, 3, 9, 11)
The 36 possible combinations: [(0, 0), (0, 1), (0, 2), (0, 3), (0, 9), (0, 11), (1, 0), (1, 1), (1, 2), (1, 3), (1, 9), (1, 11), (4, 0), (4, 1), (4, 2), (4, 3), (4, 9), (4, 11), (5, 0), (5, 1), (5, 2), (5, 3), (5, 9), (5, 11), (8, 0), (8, 1), (8, 2), (8, 3), (8, 9), (8, 11), (12, 0), (12, 1), (12, 2), (12, 3), (12, 9), (12, 11)]
And their respective sums: [(0, 1), (1, 2), (2, 2), (3, 2), (4, 2), (5, 2), (6, 2), (7, 2), (8, 2), (9, 2), (10, 2), (11, 2), (12, 2), (13, 2), (14, 2), (15, 2), (16, 1), (17, 1), (19, 1), (21, 1), (23, 1)]
Found a pair of dice that have equal probability of summing to a value from 1 to 12!
Dice 1: (0, 1, 5, 6, 11, 12)
Dice 2: (0, 1, 2, 3, 4, 10)
The 36 possible combinations: [(0, 0), (0, 1), (0, 2), (0, 3), (0, 4), (0, 10), (1, 0), (1, 1), (1, 2), (1, 3), (1, 4), (1, 10), (5, 0), (5, 1), (5, 2), (5, 3), (5, 4), (5, 10), (6, 0), (6, 1), (6, 2), (6, 3), (6, 4), (6, 10), (11, 0), (11, 1), (11, 2), (11, 3), (11, 4), (11, 10), (12, 0), (12, 1), (12, 2), (12, 3), (12, 4), (12, 10)]
And their respective sums: [(0, 1), (1, 2), (2, 2), (3, 2), (4, 2), (5, 2), (6, 2), (7, 2), (8, 2), (9, 2), (10, 2), (11, 2), (12, 2), (13, 2), (14, 2), (15, 3), (16, 2), (21, 1), (22, 1)]
"""

We can see that there are actually five unique solutions if you restrict your pips from 1 to 12:

Dice 1: (0, 1, 2, 3, 4, 10)
Dice 2: (0, 1, 5, 6, 11, 12)

Dice 1: (0, 1, 2, 3, 9, 11)
Dice 2: (0, 1, 4, 5, 8, 12)

Dice 1: (0, 1, 2, 3, 9, 12)
Dice 2: (0, 1, 4, 5, 8, 11)

Dice 1: (0, 1, 2, 9, 10, 11)
Dice 2: (0, 1, 3, 4, 6, 7)

Dice 1: (0, 1, 3, 4, 6, 7)
Dice 2: (0, 1, 2, 9, 10, 11)

But in fact there are lots more solutions (infinitely many, actually!) if you do not restrict the numbers you use in each face of the die from 1 to 12. For example, this is a possible solution:

Found a pair of dice that have equal probability of summing to any value from 1 to 12!
Dice 1: (1, 2, 3, 10, 11, 14)
Dice 2: (0, 3, 6, 11, 13, 14)
The 36 possible combinations: [(1, 0), (1, 3), (1, 6), (1, 11), (1, 13), (1, 14), (2, 0), (2, 3), (2, 6), (2, 11), (2, 13), (2, 14), (3, 0), (3, 3), (3, 6), (3, 11), (3, 13), (3, 14), (10, 0), (10, 3), (10, 6), (10, 11), (10, 13), (10, 14), (11, 0), (11, 3), (11, 6), (11, 11), (11, 13), (11, 14), (14, 0), (14, 3), (14, 6), (14, 11), (14, 13), (14, 14)]
And their respective sums: [(1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1), (10, 1), (11, 1), (12, 1), (13, 2), (14, 4), (15, 2), (16, 3), (17, 3), (20, 1), (21, 1), (22, 1), (23, 1), (24, 2), (25, 2), (27, 1), (28, 1)]

You can of course replace the numbers 13 and 14 by any other whole numbers that are larger, which would not change the result.

tl;dr of my Politics undergraduate thesis

2021-05-01T00:00:00Z

Thesis TL;DR

Abstract

How should electoral districts be drawn? In the U.S., many states attempt to limit gerrymandering by requiring that districts be "reasonably compact", but also require that plans respect "the integrity of communities of interest". Yet mandating compactness may come at the cost of communities of interest. In order to achieve a compact district shape, one may need to disregard communities of interest and assemble highly heterogeneous districts as a result, adversely affecting democratic outcomes like representation and responsiveness. Are compactness and community fundamentally conflicting goals?

I make two contributions in this work. First, I develop a new compactness metric---human compactness---that improves upon previous measures by incorporating a notion of travel times. Second, I use a Markov Chain Monte Carlo (MCMC) approach to generate a large sample of districting plans. I find mixed evidence for a trade-off between compactness and homogeneity depending on the compactness measure used. I further find that my human compactness measure consistently identifies more homogeneous districts, suggesting that a judicious choice of compactness metric can in fact encourage keeping communities of interest together.

Introduction (front part)

We care about how districts are drawn because of representation: formalistic and descriptive/substantial representation

Formalistic representation is about accountability: gerrymandering lets incumbents stay in power regardless of their performance and are thus unaccountable. Hence legislators have tried to prevent this using compactness measures

Descriptive/substantial representation is about their elected representatives fighting for their electorate's interest: this is usually best done when districts are homogeneous/ keep communities of interest together

However, one might think that compactness and district homogeneity would come apart: a perfect square might be very compact but cleave a community of interest in two, that sort of stuff. So we want to know if there's a tradeoff and my thesis addresses this.

Two key research questions

Is there a tradeoff between compactness and communities of interest?
Do some compactness measures better encompass communities of interest than others?

Methodology

Generate a large and representative subset of plausible districting plans
Evaluate compactness and spatial diversity scores on that subset of plans
Analyse the overall relationship between compactness and communities of interest (as operationalised by spatial diversity)

Human compactness and why the inter-voter distance is important

"Paramount to the idea of single-member districts is that there is great value in voters who live in the same area being put in the same district. Eubank and Rodden [2019] write:

"Voters in the same area are likely to share political interests; voters in the same area are better able to communicate and coordinate with one another; politicians can better maintain connections with voters in the same area; voters in the same area are especially likely to belong to the same social communities --- all suggest the importance of voters being located in districts with their geographic peers."

Due to the fact that voters in the same area share many things in common, A point-wise distance measure that tries to put people together in the same district could be a remedy for the key claim levied against compactness---that it splits communities of interest. A wealth of empirical evidence supports the above statement. \cite{arzheimer2012} find that constituents support less strongly candidates that live far from them, even controlling for strong predictors of vote choice like party feeling and socio-economic distance. Similarly, \cite{dyck2005} find that voters living further away from a voting site are less likely to turn out to vote. In part, voters strongly support proximate candidates because they think that these candidates better represent their interests. If voters prefer a representative who lives close to them, then we can satisfy the most voters by drawing districts where everyone lives close to everyone else---only then can that district have a representative who lives close to everybody.

In contrast, districts that put people with unrelated, faraway others carve voters out of their natural communities and are thus to be avoided. We care about whether co-districtors live in the same area and belong to the same communities of interest, not just the compactness of their electoral district. And point-wise distance metrics deliver exactly that.j

Why Euclidean distance is insufficient

"Suppose there is a city on a hill example"

Suppose there is a city on a hill. On the West side is [a] mild, long incline toward the rest of the city, which is in a plane. On the East side is a steep cliff, either impassable or with just a narrow, winding road that very few people use. While the next residential center to the East is much closer to the hilltop on a horizontal plane, it is much further on all sorts of distances that we think might matter: transportation time, intensity of social interactions, sets of shared local public goods and common interests, etc. Thus, for all practical purposes, one probably wants to include the hilltop in a Western district rather than an Eastern one. More general notions of distance can handle this.

Here we see the key problem with using Euclidean distances in point-wise distance metrics. The "impassable" region on the East would have a short Euclidean distance, and any districting plan that put the hilltop with the Eastern district would be unfairly penalised by these point-wise distance metrics. Evidently, using driving durations instead would give us more accurate scores. Using driving durations, the impassable region would have a long driving duration, accurately reflecting the political geography. In this and many other cases like it (e.g. large bodies of water), driving durations better reflect a state's unique political geographies.

NOTE: let's see if we can find these in real life

What I actually did

Use Recom neutral ensemble to generate 10,000 districting plans for each state
Calculate spatial diversity and all compactness scores
DO data analysis:
- Some initial analysis looking at New Hampshire and Idaho showing that there is a difference in the best plans under each compactness measure
- Correlation matrix showing that compacntess measures correlate with one another but human compactness less so: "this result is encouraging because it shows that these metrics are able to get at the same concept of compactness despite having completely different theoretical backgrounds".
- Did some scatterplots that basically say the same thing: compactness measures are similar but not too similar
- Spatial diversity varies a lot between states but little within states: may be due to the fact that the plans I generate are meant to resemble plans that a non-partisan districting committee would draw
- KDE plot of districts and spatial diversity: the smaller (by area) the district, the higher the diversity. Most probably an urban-rural effect.
Data analysis part 2:
- Running multivariate OLS regressions with country dummies finds no effect of different compactness measures on spatial diversity; only human compactness sees a significant negative effect
- Previously we were doing the regression with all of the plans: now consider the subset of plans:
  - I look at Top 100/90/75/25/10%, 5% plans under each compactness measure and find similar results (5% and 2% sample size too small for significance but coefficient is the same)
  - I look at the mean spatial diversity of the top plans under each compactness measure, performing a difference in means test. Only human compactness has a mean spatial diversity significantly lower than the mean spatial diversity of all plans.
  - I then perform a more robust difference in means test: I look at the mean spatial diversity of the most compact plans of each state, and compare it with the mean spatial diversity of all plans, and again, human compactness beats them all

What needs to be done next

One thing we could do is run all the correlations and difference-in-means tests with Euclidean distance human compactness. I suspect that there won't be much of a difference (possibly nothing that approaches statistical significance)
The other thing we could do is after running all of them with Euclidean distance human compactness, find states/plans where they differ a lot and see where they differ
Another thing we could do if we don't want to run it on all the plans is to pick a plan, calculate ED human compactness for each point in that plan, and compare that to DD human compactness. See if they meaningfully differ.
Either way we have to come up with the Euclidean distance metric and generate plans. If only we had a pipeline to make this faster.

MMODiscord: Run your own virtual world on any Discord server

2021-04-07T00:00:00Z

Introduction

https://github.com/juxd/discord-spatial-layer Hackathon submission link

For NUS HackNRoll 2021 (a hackathon organised by National University of Singapore) we set out to build a virtual world that can sit on top of any Discord server (working name: MMODiscord). This post will be in two parts: first, I'm going to share what we built, and then I'm going to share The Dream: what it could be, and why we would want it.

What we built and why we built it

In lieu of an actual childhood, I fondly recall hanging out and chatting with my friends in Henesys Market and fishing/cutting trees together in RuneScape's Draynor Village.

There's something uniquely magical about that sort of experience, and I'm pretty sure it's not just the nostalgia talking. They seem to better fulfil the raw and real human need for connection.

While has been solved somewhat by chat/social apps such as MSN Messenger, Telegram, and Discord, I've always felt that they never came close to the fun I had just sitting around in the FM with my friends. Useful as they are, something is missing from these apps.

What is missing? What's that je ne sais quoi? I believe it's spatiality and interactiveness. Hanging out in Maple or RS is especially compelling because it has something more than just chat. You can see your friends in the world and hang out together in a virtual space. I've written about this before. You can express yourself with your movement, fashion choices, skills, facial emotions, and so on.

So we know that game worlds have advantages. They're more spatial, more interactive, more immersive, more compelling. So what if we could get the best of both worlds? What if we had a spatial "layer" that sits on top of the social apps we know and love?

The "spatial layer" we built has the following features:

Connects seamlessly to your Discord account
Move around and chat
Chat appears above your head
Use number keys 1-5 to change your mood (corresponds to MapleStory)
Available both as a standalone webpage and as a Google Chrome extension (see submitted links)

My thinking:

How to use

We came up with a custom authentication solution:

Type !joinspace into the server you'd like to; our bot DMs a secret to the user
Navigate to our GitHub Pages link here IF ON FIREFOX type in about:config in the URL bar, and set network.websocket.allowInsecureFromHTTPStotrue`
Enter secret into the GitHub Pages link
You have now joined the spatial layer and can move around, chat and use emojis.

Technical details

We used a couple of libraries:

Discord API for the Discord bot
Nengi for the multiplayer networking
PixiJS for the rendering engine

We built the server auth flow ourselves. The primary challenge is how to match the Discord user with the person who joins the game, so when you type !joinspace our bot will DM you a secret (e.g. "chubby-flappy-bird"). The server knows which secret it's given to which Discord user, so when you enter the secret and join the server we can then assign you the right avatar.

Difficulties and things we've learned

Dexter: "I wasted a day on a Chrome Extension"
Vignesh: "Bro it's a hackathon there's nothing to think you just do"
Julius: We adapted well but we could have a better job at understanding what we wanted.

Why would anyone use this? What is the point of it all?

There are lots of startups in this space like Gather Town, but (in my opinion) they won't get much traction because they aim to be a tool, not a community. We want to build something that piggybacks on--supplants--the communities already existing in groups like Discord and Telegram.

Why not just play a game instead? Our spatial layer is really lightweight and doesn't require a download: good luck getting the 1,000 people on your server to download 20GB of game.

Extensions

We've built a base of movement and chat, the two most important features. We want to give more activities for users to do and increase Discord integration. For instance, we want to allow users to play games with Discord bots like catch Pokemon, farm, fish, etc. We want to build persistent user and server storage. We also want to allow each server to place objects in and customise the look of their server.

We might want to give players of this gameworld something to do: the equivalent of Maki Bot in Discord servers.

Different servers would have different tilesets (like server-specific emoji) and construct their gameworld differently. One server might use a generic tileset to build a massive mediaeval castle while another might use a cyberpunk-inspired tileset to build Night City. This would allow for easy customisation of the space.

It would be really cool if we could allow fellow HnR participants to code their own equivalent of Bots: maybe expose some API where we allow squares and players to have arbitrary metadata associated with them, and allow getters and setters on that arbitrary metadata.

At the most basic and boring level, for instance, a bot could be written to assign players a "Fame" score (just an integer). And players could interact by faming or defaming other players. But more complicated interactions could exist. What if a bot assigned each square a terrain metadata type and you could fish on squares with terrain:water? This would be an API call to a Discord bot which would then update your player's inventory metadata field to have a random fish? What if you wrote a bot that could trade items between players? There are tons of possibilities.

Why not just play an MMO? Or use Second Life?

(WIP section)

A discord server is already a community and you want to build something on top of it, not try to transplant/supplant the existing community
zero friction is important! how can you possibly convince everyone in your server to move to a game
very lightweight layer not a entire game is a feature
the layer is completely optional so you can still have the same community even when not in the game

Other stuff we could do

This is an instantiation of the following meta-level principle of building worlds for hyperlocal communities.

meta-level: build X ( tinder, facebook, twitter, a game world, etc... ) for a hyperlocal community (the HnR discord server, Telegram group)
meta-meta-level: build Y that allows us to build some X for any hyperlocal community

I like the meta idea of building something over the hyperlocal community. Weineng suggested something like Tinder or FB but only for HnR participants.

Some ways to deal with big(-ish) data: 'why do we get an OutOfMemory error in FPGrowth?'

2021-03-31T00:00:00Z

(Written for work)

Introduction

Xin Zheng mentioned that she was running out of memory when running the a priori and FP Growth algorithms for the recommender systems project.

After some research, I found the reason for the error, and here I present a few solutions which might help other data scientists too.

Thanks also to Siyang for being a sounding board for my ideas.

The problem

Xin Zheng mentioned that she was running out of memory when running the a priori and FP Growth algorithms These algorithms are used to generate frequent itemsets (items that are frequently bought together by customers) so that we can recommend similar items to purchase.

This is the section of her code that was throwing the error:

"""
FP growth algo for product association rules calculation
* input item order-product name matrix
* output rules
"""
import pandas as pd
from mlxtend.frequent_patterns import fpgrowth
from mlxtend.frequent_patterns import association_rules
import time

class FPGrowthRules:
    def run(self, df_matrix=None, min_support=0.5, min_confidence=0.25):
        frequent_itemsets = fpgrowth(df_matrix, min_support=min_support, use_colnames=True)
        # ... stuff elided ...
        return rules

In particular, it was this invocation of fpgrowth from the mlxtend package:

frequent_itemsets = fpgrowth(df_matrix, min_support=min_support, use_colnames=True)

Here fpgrowth takes in a DataFrame and returns a DataFrame. In particular, it gets frequent itemsets for a one-hot DataFrame. A one-hot DataFrame looks like this:

        Apple  Bananas   Beer  Chicken   Milk   Rice
    0   True    False   True     True  False   True
    1   True    False   True    False  False   True
    2   True    False   True    False  False  False
    3   True     True  False    False  False  False
    4  False    False   True     True   True   True
    5  False    False   True    False   True   True
    6  False    False   True    False   True  False
    7   True     True  False    False  False  False

fpgrowth returns a pandas DataFrame with columns ['support', 'itemsets'] of all itemsets that are >= min_support and < max_len. You can find usage examples here.

After looking at her code and the source code of the mlxtend library, I found the error and came up with three plausible solutions, each having their own set of advantages and disadvantages.

Solution 1: use a bigger machine (vertical scaling)

The simplest way to solve the out-of-memory issue is to use a larger machine. XZ's machine (27GB of RAM and 10GB of disk) runs out of memory, but 27GB is not that much memory in the grand scheme of things; we can get instances with much more memory.

The pros of this solution is that it requires no developer effort. The cons are that this will incur additional cost, and the TP we pass it to may not be able to spin up larger machines. And of course, this solution is not very scalable because there's a hard cap on the amount of memory we can provision. So if the out-of-memory issue is caused by some $O(n^2)$ scaling or worse, this would not be a good solution.

Solution 2: use Apache Spark (horizontal scaling)

One more complicated way is to use Apache Spark to scale out. This involves rewriting Xin Zheng's code as a Spark job and using a parallel, Spark-friendly version of fpgrowth. We would then find some way to spin up a Spark cluster and run the Spark job.

The pros of this solution is that it can scale to much larger datasets. The cons are that it introduces a lot of technical complexity both in the code, and in the deployment. It will also make testing more difficult. Additionally, TPs may not have the technical chops to provision their own spark cluster.

Solution 3: replace mlxtend's source code (software solution)

The last solution requires us to actually understand the problem. Let's figure out exactly what is causing the OOM error.

Recall that fpgrowth takes in a one-hot DataFrame that looks like this:

        Apple  Bananas   Beer  Chicken   Milk   Rice
    0   True    False   True     True  False   True
    1   True    False   True    False  False   True
    2   True    False   True    False  False  False
    3   True     True  False    False  False  False
    4  False    False   True     True   True   True
    5  False    False   True    False   True   True
    6  False    False   True    False   True  False
    7   True     True  False    False  False  False

and returns a pandas DataFrame with columns ['support', 'itemsets'] of all itemsets that are >= min_support and < max_len. It's clear that the smaller min_support is, the more itemsets will be returned. If min_support is small enough we will get an OutOfMemory error.

But how are these itemsets generated? Can we pin down the exact line that is causing the error? How do we fix the OutOfMemory error?

Let's dive into the fpgrowth source code here:

We can see that the fpgrowth function calls fpc.setup_fptree on line 84 to create a class called an FPTree, then uses this tree class to run fgp_step recursively.

I did some additional digging and found the most likely cause: line 113 inside fpg_step:

# fpg_step
if max_len:
    size_remain = max_len - len(tree.cond_items) + 1
for i in range(1, size_remain):
    for itemset in itertools.combinations(items, i): # <-- this line!
        count += 1
        support = min([tree.nodes[i][0].count for i in itemset])
        yield support, tree.cond_items + list(itemset)

Here itertools.combinations(items,i) is almost certainly causing the OOM error.
$items \choose i$ where items and i are large will give a very big number, on the order of $\Theta(items^i).$ (see the Appendix for an explanation). So itemsets grows non-polynomially with the size of the DataFrame. But this itself is not the problem, because itertools returns a generator, which will not OOM. Rather, it's the line immediately after it, fpc.generate_itemsets, that is the issue:

# fpgrowth.py
# ...
def fpgrowth(...):
    # ....
    generator = fpg_step(tree, minsup, colname_map, max_len, verbose)
    return fpc.generate_itemsets(generator, len(df.index), colname_map) # <- this line

# fpcommon.py

def generate_itemsets(generator, num_itemsets, colname_map):
    itemsets = []
    supports = []
    for sup, iset in generator:
        itemsets.append(frozenset(iset))
        supports.append(sup / num_itemsets)

    res_df = pd.DataFrame({'support': supports, 'itemsets': itemsets})

    if colname_map is not None:
        res_df['itemsets'] = res_df['itemsets'] \
            .apply(lambda x: frozenset([colname_map[i] for i in x]))

    return res_df

Looking at the generate_itemsets function we can see that itemsets and supports are appended to for each itemset. And we know that there are a lot of itemsets. This causes the OOM error.

So what can we do? There are several possible solutions. First is simply to use a higher min_support value so that we don't run out of memory. But this is of course not a solution so much as it is giving up. I think one real solution is to generate the DataFrame in chunks, and persist it in disk. This way we don't have to append all itemsets to itemsets before generating the DataFrame.

What are the pros and cons of this solution? This solution does not need any additional hardware or cost: this is a pure software solution. This would also be the easiest to hand over to the TP, as it introduces no additional technical complexity.

The con is that this would require us to make a pull request to the mlxtend repo, which may or may not be accepted. We must also find a way to make the returned chunked DataFrame compatible with just a regular, non-chunked DataFrame. But if accepted this would be a public service to others using the repo too. I know Siyang has made PRs to an open-source library in the past that have been well-received by the community.

Conclusions

I think one key takeaway here is to understand the problem domain well, and not to be afraid to dig into the source code. Knowing what we now know about the problem domain, it's clear that solution 1 would not be sufficient as the output size explodes with an increase in input size. Solution 2 does work but involves significant technical complexity. In my opinion solution 3 is probably best.

We could also sidestep the problem entirely: reduce size_remain or items by setting a larger min_support threshold. I know this is what XZ has been working on. This does come at the expense of accuracy/not having enough recommendations, however.

Acknowledgements

Thanks to Siyang and XZ for telling me about this problem.

Appendix

items and i grow as min_support shinks.

We can see this by looking at the source code for setup_fptree (here). The line is items = np.nonzero(item_support >= min_support)[0].

Notes on AWS's serverless framework

2021-03-24T00:00:00Z

(written for work)

Introduction

On Hui Lim's recommmendation, I'm working through the following courses to familiarise myself with the AWS Serverless framework:

I'm learning a lot from these courses and I don't want to forget this stuff, so I'm writing it down as notes.

Maximilian's course covers the following:

API Gateway
AWS Lambda
DynamoDB
AWS Cognito
S3
AWS Route53

A lot of the course focuses on manual setup of the API Gateway and DynamoDB via the web portal which is of course automatable by AWS's CLI. But going through the UI and doing the manual setup actually helps a lot to understand the "big picture": the structure of the entire AWS ecosystem, the API gateway, and so on.

Ariel's course covers some of the same ground of Maximilian's. I'll be working through it very shortly and writing my notes here too.

API Gateway/AWS Lambda

Alex de Brie's "A Detailed Overview of AWS API Gateway is a great overview, and is the source of the following image:

There are six main steps in the AWS Gateway request-response cycle.

Authentication
Method Request: in this step we validate the incoming request to make sure it corresponds to a particular JSON schema. This simplifies the backend/error handling code.
Integration Request: transforms the client's request into a shape that the server/Lambda function can read.
Integration: the API gateway sends the request to the backend server/Lambda function, which receives the request and does stuff with it (sends an email, reads/writes a DB, performs a calculation...).
Integration Response: this replaces the result of the lambda function with a function that the API Gateway can receive.
Method Response: this returns a response with a proper HTTP response code.

The API Gateway's request-response cycle in detail

Six steps, five if you ignore the actual integration because it's not part of the API Gateway:

Authentication
Method Request
Integration Request
(actual integration -- lambda function or other backend service)
Integration Response
Method Response

The first gatekeeper is authentication: this can use a custom (lambda) authoriser, or use Amazon's Cognito authoriser. More on this later.

The Method Request step validates the incoming request to make sure it adheres to a particular model (has particular fields, etc).

If it does then we move to the Integration Request: transforming incoming data into a shape we want to use on the action we're about to trigger. A transformation can be done using the Velocity Template Language (VTL) to add headers, metadata, auth info, and so on. One good usecase for this is in retrofitting a new JSON API to an existing older backend. For example, your server might accept only application/xml while your requests are in application/json. This is where you would translate your incoming request inoto a format the server can read.

Then of course you move on to the actual integration step, where a backend (or Lambda function of your choice) is run. This Lambda function

Integration response: this is the return of the lambda function, resolved calculation/db look up etc.

Method Response shapes the response received by the client

What does it mean to "configure as proxy resource"?

Means catch-all: catch all requests.

If you catch all incoming requests, you can create your own router

What is "API Gateway CORS"?

CORS: cross origin resource sharing.

Imagine your SPA runs on example.com and the API runs on api.com. When the SPA tries to ask for a request from api.com, the browser will prevent this request from being handled, so api.com needs to send some "preflight headers" that will convince the browser that it's OK (but what if api.com is malicious?) before this cross-domain request will be allowed.

"Access-Control-Allow-Headers" allow
"Access-Control-Allow-Origin": all domains may send these requests and therefore we're good to go.

What is "Lambda Proxy Integration"?

use incoming metadata like headers and authentication etc to pass that unfiltered data over to the lambda

This also means that the Lambda function will be responsible for handling the authentication rather than the integration request

How do I see what the integration receives/logs?

Use AWS Cloudwatch.

What are "body mapping templates" and why would we use this?

We can use body mapping templates to transform the result of a lambda function into a more standardised form. Why would we do this? If you have a lambda function that returns a value but your client is expecting a particular response schema, you can use the body mapping template inside the integration response step to transform it.

Similarly in the integration request step we can also use this template language to edit the request body your integration receives. I've given an example above:

What are "variable path parameters"?

Variable path parameters allow you to run the same Lambda function with different input parameters (path parameters). For example, GET request to /all vs GET request to /single can be handled with the path parameter on resource /{type} and then in the Integration Request we can get this path parameter by doing $input.type.

You can use query parameters too ?someParam=someValue; use them for validation and extract them with templates.

How does AWS Custom authorisation work?

We can create a custom authoriser (also called Lambda authoriser) on the Method Request step.

The custom authoriser is a Lambda function that takes in some information from the incoming request (specifically, an authorisation token). That function will run some code to validate/identify that user, and returns an IAM policy to us + a user ID + optional "context".

If the auth token is missing or invalid the IAM policy will give a rejection via a Gateway Response. A Gateway Response is a "short-circuit" response returned by the API Gateway; it's a response that is returned before the request hits the backend.

If it is a success then this 3-tuple will then be passed it into the next step, the Method Request step. (? Is it the Method Request step ?) "This context will be added to the event object in your backing Lambda function"

API Gateway Lambda authorization workflow

source: AWS docs

First, the client calls a method on an API Gateway API method, passing a bearer token or request parameters. The API Gateway checks whether a Lambda authorizer is configured for the method. If it is, API Gateway calls the Lambda function.

The Lambda function authenticates the caller by means such as the following:

Calling out to an OAuth provider to get an OAuth access token.
Calling out to a SAML provider to get a SAML assertion.
Generating an IAM policy based on the request parameter values.
Retrieving credentials from a database.

If the call succeeds, the Lambda function grants access by returning an output object containing at least an IAM policy and a principal identifier.

API Gateway evaluates the policy.

If access is denied, API Gateway returns a suitable HTTP status code, such as 403 ACCESS_DENIED.
If access is allowed, API Gateway executes the method. If caching is enabled in the authorizer settings, API Gateway also caches the policy so that the Lambda authorizer function doesn't need to be invoked again.

Gotchas

What do I do if I get an "Unsupported Media Type Error" on the client side?

Make sure to set content-type to application/json rather than text/plain

Things to learn

How do IAM policies work? What happens when the Lambda authoriser returns an IAM policy?

IAM policy docs

How do CORS headers work?

What's a Swagger definition?

What's the difference between the Integration Response step and the Method Response step?

Integration responses transform the response from the backend lambda function (which can be any value at all) into a response that API Gateway can handle.

Method responses validate the output to the client. You must first create your method repsonses before you can use a given status code in an integration response. (source: Alex de Brie)

You can use a VTL template to transform the response in both the integration response step and the method response step (? is this true?)

DynamoDB

What is a Partition Key?

A partition key must be unique.

DynamoDB allows you to use a partition key PLUS a sort key as a primary key, where the combination has to be unique.

Why is it called a partition key? Because these keys are assigned to different partitions. (Are they hashed?)

What is a Global Secondary Index? What's a Local Secondary Index?

What is read and write capacity?

You provision certain read and write capacity units: kilobytes read/written per second. You can also just use on demand.

What's the difference between `scan()` and `getItem()`?

scan() returns an array, getItem returns only one item.

Gotchas

Why do I get an Access Denied Exception?

You haven't set permissions right: you need to give your lambda function the correct DynamoDB policy.

AWS Cognito

Cognito flow

Initialise UserPoolId and ClientId.
Create a new CognitoUserPool. It is this object that will be used behind the scenes.
Create a new attribute object (standard attribute)
Create a CognitoUserAttribute array
Call the userPool.signUp method with username, password, attributeList, and callback function that gets executed once Cognito is done.
Cognito should actually send you an email and you should be able to verify.

After signup:

Initialise a CognitoUser object.
Call CognitoUser.authenticateUser() with AuthenticationDetails and a callback function.

What is a Cognito User Pool?

Amazon Cognito user pools documentation

A user pool is a user directory in Amazon Cognito.

What's an App Client?

App clients have access to the user pool. Neither user pool ID nor app client ID are secrets.

AuthenticationDetails object

Should contain username and password

CognitoUser object

should be initialised with a username and userPool.

CognitoUserSession object?

CognitoUserSession will give you JWTs: CognitoAccessToken, CognitoIdToken, and CognitoRefreshToken.

authIsLoading

What's the difference between CognitoAccessToken, CognitoIdToken, and CognitoRefreshToken?

TODO

this.getAuthenticatedUser().signOut()

will actually delete the localstorage tokens

this.authService.getAuthenticatedUser().getSession()

checks if tokens are still valid

CognitoIdentityServiceProvider

Why would you use query string parameters to pass tokens?

Rather than passing it in the body request/headers? It seems like that would be bad because this would allow MITMs to use your token

Gotchas

Why do I receive a empty userId when using Cognito authoriser?

When using Cognito user authoriser, our existing body mapping template will not get the userId with $context.authorizer.principalId. Instead, we need to use $context.authorizer.claims.sub to get the userID.

Security

Acknowledgements

Thanks to Hui Lim for sending me these excellent courses.

Bibliography

Singh, 2020. Creating an API Gateway Lambda Authorizer
Alex de Brie's "A Detailed Overview of AWS API Gateway is a great overview, and is the source of the following image:
de Brie. Lambda custom authorizers.
Green 2018

Notes on Dynamo (2007)

2021-03-20T00:00:00Z

Introduction

I read the following the paper as part of my self-study of 6.824 with NUS SoC folks. It may or may not be part of my goal to finish a Stanford CS degree in a year.

This week we discussed the paper DeCandia et al. (2007). Dynamo: Amazon's Highly Available Key-value Store

Some useful links are the 6.824 lecture notes and the 6.824 lecture FAQ

The big picture

Dynamo is trying to build a distributed key-value store focusing on write availability and write latency.

Motivation: Amazon doesn't need a RDBMS for many of its services, only a key-value store: "There are many services on Amazon's platform that only need primary-key access to a data store... for many services... the common pattern of using a relational database would lead to inefficiencies and limit scale and availability."

So we have a more relaxed operations model: we only require getting and setting keys. However, there are two things we care about a lot:

Firstly, Amazon cares a lot about latency, because any weak link in the chain will result in cascading latencies on the rest of the services.

Secondly, we want it to be always writeable despite failures. In traditional systems, "writes may be rejected if the data store cannot reach a majority of the replicas at a given time." This is unacceptable to Amazon: e.g. you can't have Amazon reject adding items to shopping carts. Here Amazon allows you to tweak the parameters: if you always want writes to succeed, you can set W to 1, which ensures that a write is accepted as long as a single node in the system has durably written the key to its local store. "However, in practice, most Amazon services in production set a higher W to meet the desired level of durability."

Also, we don't care about consistency so much: "data stores that provide ACID guarantees tend to have poor availability." Instead DynamoDB guarantees eventual consistency.

Big ideas

Consistent hashing allows incremental scalability

Dynamo uses consistent hashing, which allows nodes to be added and removed on-demand very quickly. Let's see how it does this.

Why do we need to hash?

Before talking about consistent hashing, let's talk about hashing. Why hash? We hash in order to allocate keys evenly to nodes. Each node is responsible for only a small set of keys (called its range), because the number of keys is too large for any single node to hold.

When a node receives a read or write request for a key, it will hash it, then check if that hash is in its range. If it is then it will service that request. If it receives a request for a key not in its range, it will forward the request to the correct node. (How does it know who to forward the request to? More on this later.)

What is consistent hashing and how does it allow for incremental scalability?

OK, so that's hashing. What about consistent hashing? The idea of consistent hashing is to simply wrap the range around, so the largest value wraps around to the smallest value. Wikipedia says that consistent hashing is a special kind of hashing such that when a hash table is resized, only a few keys need to be remapped on average.

Why is this necessary? We'll soon see that consistent hashing allows incremental scaling in the sense that adding a new node only changes the keys of N nodes (where N is small -- 3), while using non-consistent hashing would require all keys to be remapped on average.

How do we use consistent hashing?

So in consistent hashing the key space is mapped to a big circle where the largest hash value (in this case, $2^128 - 1$ ) wraps back to zero.

Note that consistent hashing is a different thing from replicating the data on multiple nodes. Consistent hashing allows for replication and scalability, but you can use consistent hashing without replication.

How are nodes assigned to keys

Each node is randomly assigned a point on the consistent hashing circle, and it's responsible for all points between it and the previous node on the circle.

The above figure illustrates. Here the key k1 lies between the node to its left (not labeled) and the node to its right (A). So A is responsible for key k1.

How does consistent hashing allow scalability?

Recall that nodes are assigned to a key range using random assignment. What happens when we have to add or remove a node? If we remove a node then we must make sure that that node's key range is somehow given to the remaining nodes. And conversely if we add a node then we must "steal" some keys from all the nodes to give to the new node. Consistent hashing allows us to do this affecting the minimum amount of nodes.

Let's compare what happens when we add a node under consistent hashing versus non-consistent hashing. Suppose you had a 3-bit hash function hash(key) that mapped keys uniformly to an integer in range[0, 7] inclusive. And suppose that you had three nodes so each hash(key) mod 3 would give a key of 0, 1 or 2. Let's look at how each of the keys maps in mod3:

Now suppose a new node enters the cluster The key insight is that almost all of the hashes have changed (I've marked the keys that have changed with asterisks):

0 : 0
1 : 1
2 : 2
3 : **3**
4 : **0**
5 : **1**
6 : **2**
7 : **3**

This will mean that a vast majority of data (those which mapped to 3--8) has to be moved from one cluster to another. In fact, if N is small and the domain of the hash function, K, is large such that $N << 2^K$ , almost all of the data will have to be moved. The same is true if we remove a node from the cluster.

Consistent hashing avoids this problem. In the basic version of consistent hashing each hash is assigned to a node with the first ID larger than it, and each node is randomly assigned a key value. Again we'll use the three-bit hash function with three machines. Here's a very badly-drawn circle with the three nodes denoted with asterisks. In this case the three nodes have been assigned 0, 2 and 5 respectively.

     *0
  7     1
6        *2 
  *5   3
     4

0 : 2
1 : 2
2 : 5
3 : 5
4 : 5
5 : 0
6 : 0
7 : 0

Now let's insert a new node with value (say) 4:

     *0
  7     1
6        *2 
  *5   3
     *4

0 : 2
1 : 2
2 : **4**
3 : **4**
4 : 5
5 : 0
6 : 0
7 : 0

As we can see, only the nodes in between 2 and 5 need to be moved.

In general, using consistent hashing means that adding or removing a new node only requires minimal inter-node communication/key range changes, which allows for scalability.

How does consistent hashing allow replication?

In order to ensure durability/availability, you can't have only one node save the data. The data must be replicated over multiple nodes: N nodes, to be exact.

Figure 2 illustrates how this is done. The N-1 nodes clockwise of the key range replicate the keys of that node. The keys that fall within (A, B] are the purview of B. The two other nodes C and D that lie clockwise of B will replicate this key range too.

There's the idea of a preference list: a preference list is the list of physical nodes responsible for storing a particular key. (We'll see why I mention the word physical later.)

Having virtual nodes (tokens) improves on consistent hashing

Section 4.2 of the Dynamo paper:

The basic consistent hashing algorithm presents some challenges. First, the random position assignment of each node on the ring leads to non-uniform data and load distribution. Second, the basic algorithm is oblivious to the hetereogeneity in the performance of nodes. To address this issues... instead of mapping a node to a single point in the circle, each node gets assigned to multiple points in the ring... Dynamo uses the concept of "virtual nodes". A virtual node looks like a single node in the system, but each node can be responsible for more than one virtual node.

Why is this an improvement? The MIT Aurora FAQ says:

They are a scheme to balance the keys nicely across all nodes. If you throw N balls in B bins, you get on average B/N balls in a bin but some bins may have many balls. To ensure that all bins are close to the average, you can place multiple virtual bins in each real bin. The average of multiple random variables has lower variance than a single random variable.

How does one assign nodes to keys?

Amazon actually iterated over three versions of assigning nodes to keys.

Vector clocks and hinted handoff allows for an always-writable database

The next big idea is to allow reads to return not just a value, but a tree of possibly-inconsistent values. This is in effect giving up the consistency property of "reads must observe the latest write": now reads may observe lots of different writes, one of which is the latest. If you allow this then you can make your database always writeable, which is Amazon's explicit design goal. We'll also see how nodes make sense of the different writes.

So what exactly happens in a read/write?

Read and write operations involve the first N healthy nodes in the preference list, skipping over those that are down or inaccessible. Whenever a node receives a read or write request on an object, it will increment its own sequence number. For a get() request, the coordinator requests all existing versions of data for that key from the N highest-ranked reachable nodes, then waits for R responses before returning the result to the client. If the coordinator ends up gathering multiple versions of the data, it returns all the versions it deems to be causally unrelated. The divergent versions are then reconciled and the reconciled version superseding the current versions is written back. Note that with eventual consistency, it's not guaranteed that the data it gets back reflects all the writes!

For a put() request, the coordinator generates the vector clock for the new version and writes the new version locally. In Dynamo, all objects are immutable, so the only way to change an object is to generate a new version. It then sends the new version (along with the new vector clock) to the N highest ranked reachable notes. If at least W-1 nodes respond then the write is considered successful.

How vector clocks are used in reads/writes

So what's a vector clock? A vector clock is effectively a list of (node, counter) pairs. List[Tuple[Node, Counter]] One vector clock is associated with every version of every object. (Objects are immutable). One can determine whether two versions of an object are on parallel branches or have a causal ordering by examining their vector clocks.

How do we use vector clocks to determine if two versions of an object are causally ordered or are diverging branches? "If the counters on the first object's clock are less-than-or-equal to all of the nodes in the second clock, then the first is an ancestor of the second and can be forgotten. Otherwise, the two changes are considered to be in conflict and require reconciliation."

Handling failures

Hinted handoff

Dynamo allows writes to return to the caller before the update has been applied at all the replicas: "Upon receiving a put() request for a key, the coordinator generates the vector clock for the new version and writes the new version locally. The coordinator then sends the new version (along with the new vector clock) to the N highest-ranked reachable nodes. If at least $W-1$ nodes respond then the write is considered successful."

Note the word reachable here. This is what hinted handoff is: some less-preferred nodes will help keep some data meant for the preferred, unreachable node. and try to periodically re-deliver it to the preferred node which is temporarily down. Once the transfer succeeds the less-preferred node will delete the object from its local store. This is used for temporary node failures.

This gives us the eventual consistency property any read request will eventually read all of the values that were written because the less-preferred nodes will deliver it to the preferred nodes as long as fewer than W nodes fail.

But note that hinted handoff loses consistency. See the FAQ for this: in particular, "Why is Dynamo only eventually consistent?"

Replica synchronisation

Basically Dynamo compares nodes using Merkle trees, which allows log(N) comparisons in the average case.

Membership and failure detection

Used for permanent node failures. An administrator manually adds and removes. Not that interesting IMO.

How this relates to other database designs

Dynamo vs Aurora

The biggest difference is that Aurora is a RDBMS, Dynamo is a key-value store. Although it seems to me that Aurora could easily be made into a kv-store as well; since "the log is the database", just restrict the set of transactions to be only get and put.

In Dynamo every node is capable of receiving a write, while in Aurora there's only a single write-capable replica. This may have scalability implications, since Aurora is fundamentally limited to one writer.

In Aurora each storage segment stores 10GB segments of the log; here each node stores a partition of the entire key space. Aurora supports a storage volume up to 64TB; this is pretty big, but Dynamo can do even bigger. Since each node has its own storage, the storage limit its limited only to the key space. The key space is 128-bit. 232 is 4GB, 264 is 18.4 exabytes, and 2**128 is a ridiculous number.

Dynamo vs Memcached

Memcached and Dynamo are both key-value stores. But they solve very different problems. Memcached is trying to make reads very fast: Dynamo is trying to make writes very fast (and always available).

Memcached's writes are going to be very slow because the writes have to propagate to all the regions (? is that true?).

Again there may be differences in storage: for memcached there must be a single database instance and that probably bounds the number of keys you can have, whereas for Dynamo you could have loads of keys since each node only holds a small subset of keys.

FAQ

What are the three different partitioning strategies and why is the third one best?

This is the trickiest part of the paper, IMO. So here we're talking about how to assign nodes (both real and virtual) onto the hash space. And there are three approaches.

The first version and the one that's explained in section 4.2 is that each node is assigned several virtual nodes (tokens) and each token is randomly placed on the entire hash space. Each token is responsible for the keys between it and the token anticlockwise of it. We call this a key range.

But there's a problem with this. Section 4.7, Replica synchronisation:

The disadvantage with this scheme is that many key ranges change when a node joins or leaves the system thereby requiring the tree(s) to be recalculated. This issue is addressed, however, by the refined partitioning scheme described in Section 6.2.

Let's explain this a little bit more. Let's take a look at Strategy 1 in the diagram. Let's call the node to the left of A node O. When say node A is removed, the new key range will stretch from O to B. And that means that node C has to change its key range, looking up all the new keys individually and grabbing the keys and so on. This takes a long time. As the paper writes, the fundamental issue is that adding and removing nodes changes the data partitioning. So let's try to decouple these things.

The second approach is a middle-of-the road approach. You place the tokens exactly where you would in the first version, (random assignment). But the difference is in how you assign keys to tokens. Here the key ranges never change: instead, the key space is divided into equal-size partitions and these partitions are then assigned to tokens. A key is assigned to a token by walking from the end of its partition. This approach is better because when a node leaves or is added the key ranges don't change, so you can actually copy and move over partitions wholesale from one node to another. Also, you can simply add more nodes without changing the key range, thus decoupling the partitioning from the number of nodes.

The third approach is actually really simple. Again you divide the key space into equal-size partitions, but the difference here is that instead of placing tokens
randomly along the ring, here you place tokens uniformly (but not randomly!) along the ring. In fact, you assign Q/S tokens to each node, where $S$ is the number of nodes. And when nodes are added or removed you simply randomly distribute tokens to or from the nodes such that each node maintains $Q/S$ tokens. So instead of transferring key ranges from node to node, you simply transfer token ownership from node to node (which of course transfers the attendant tokens as well).

What are the parameters R, W, and N? Assuming there's no "hinted handoff", how many node failures can the system tolerate while still ensuring that successful read operations return up-to-date data?

N is the number of nodes any data item is replicated to. "In addition to locally storing each key within its range, the coordinator replicates these keys at the N-1 clockwise successor nodes in the ring. This results in a system where each node is responsible for the region of the ring between it and its Nth predecessor".

R is the minimum number of nodes that must participate in a successful read operation.

W is the number of nodes that must participate in a successful write operation.

"hinted handoff": a node stores some data that isn't meant for it, but keeps a record who it's meant to be for, and will deliver the replica to the intended recipient once the recipient has recovered.

Suppose we are looking at the following sequence of events:

get() call
put() call
get() call <--- successful, but must return up-to-date data.

I think the answer to the question depends on when the nodes fail. If the nodes fail after the successful put call, then as little as $W$ nodes need to fail (where $W < N-W$ ) in order for subsequent successful get() calls to no longer be up-to-date. In the worst-case scenario, all W nodes that held the latest version of the object written to have failed, and you lose your updated data (until you can bring it up again).

Call this the durability condition.

If however the nodes fail before the put call (or even before the first get call), the maximum number of node failures the system can tolerate is

min(N-W, N-R).

Call this the availability condition.

Why is this? Suppose there is no hinted handoff and we have consistent data replicated over N nodes. In order to perform a successful read there must be at least R nodes participating. So the maximum number of nodes that can fail and still allow a successful read is N-R.

Suppose we have a successful read and we now want to write the data. In order to perform a successful write we must have at least W nodes participating. So the maximum number of nodes that can fail and still allow a successful write is N-W.

And if you have no more nodes failing after the put() call, it is guaranteed that the W nodes that have successfully written the updated data are the highest W nodes in the priority list, which means that they will be selected in the next get() call, so you are guaranteed to get the latest version of the data.

The interesting thing here is that there's a fundamental trade-off between durability and availability. You can increase $W$ and decrease the likelihood of a failure of the first kind, but increase the likelihood of a failure of the second kind.

What's "eventual consistency", and what are some advantages and disadvantages? What's a specific example of behaviour that could happen under this consistency model?

Eventual consistency means that the database eventually reaches a valid state. I understand this as meaning that each of the N nodes must eventually hold the same copy of every key. More formally, there exists some time t in the future where $t > t_0$ such that for each key in each of the N nodes $key_t$ , each key's value is the same: that is, $I(key_t) == J(key_t)$ for each $(I, J) \in N$ .

Advantages: you get asynchronous propagation. put() calls are always available (provided you have at least W nodes), subsequent get() calls can return without waiting for put() calls to replicate themselves over N nodes, which is good if $W << N$ or if we have server outages.

Disadvantages: subsequent get() calls are not guaranteed to return the latest version of the data, and reconciliation must be performed.

Specific example of behaviour that could happen under this consistency model: deleted items resurfacing in a "delete item from cart" if one chooses to merge the two different objects.

How does "consistent hashing" (compared to, say, ~hash(key) % N~) reduce the amount of data that needs to be moved when a node joins or leaves the cluster?

See front bit for the full explanation.

Where is the configuration information stored? How is it kept consistent?

It's kept by each node and kept consistent by gossiping.

Why is Dynamo only eventually consistent? How can divergent histories form even with a 3-2-2 quorum setup?

It would seem on the face of it that Dynamo can be completely consistent and histories can never diverge if one uses a 3-2-2 setup. Any write must have a quorum of two, and regardless of which two you choose (AB, BC, AC), they must have at least one node in common with the latest write, right? So how can inconsistency occur even in the presence of network partitions? The key (thanks Bernard) is that hinted hand-off loses consistency!

Recall that hinted hand-off allows nodes that are not in the first N nodes of the preference list to receive and temporarily store write requests for a node that's temporarily unreachable.

But this of course introduces inconsistency. Suppose I make a write request. I walk down the preference list to find the first two reachable nodes. Suppose I get quorum from AB. The vector clock looks like this: [SA, 1].

Then I make a second write request, and let's say this time A was unreachable so I get quorum from BC instead. C is updated with B's latest version of the data, so everything is consistent. We get ([SA, 1], [SB, 2]). The second vector clock subsumes the first so when A receives this new vector clock it knows to override its own.

But what if during the second write request A and B were both unreachable? Then I would reach C, and by hinted handoff D would take over keeping a message for A. C would be holding a vector clock ([SC, 1]) and A would be holding the vector clock ([SA, 1]). Now when A reconciles it has two different vector clocks and no way to resolve them as they have nothing in common.

Questions in the lecture notes

source

Suppose Dynamo server S1 is perfectly healthy with a working network connection. By mistake, an administrator instructs server S2 to remove S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while for the membership change to propagate from S2 to the rest of the system (including S1), so for a while some clients and servers will think that S1 is still part of the system. Will Dynamo operate correctly in this situation? Why, or why not?

source

Don't see why not: when S1 node is removed, S1 should offer and upon confirmation from the nodes clockwise to it transfer the appropriate set of keys. Call these nodes say S3, S4, and S5 given N=3.

One issue is if some nodes who think that S1 has been deleted contact the nodes in the new preference list who think S1 has yet to be deleted. So a node who thinks S1 has been deleted will contact S3, S4 and S5 for a write/read request. S3 and S4 will acknowledge and agree. However, S5 will believe it's not in its key range and will return false. But here there's still a quorum of 2 out of 3, and so there will be no problem in this scenario. Eventually the node removal will be gossiped to all alive nodes.

How does Dynamo recover from permanent failure -- what is anti-entropy using Merkle trees?

A: In this paper, anti-entropy is a big word for synchronizing two replicas. To determine what is different between two replicas, Dynamo traverses a Merkle representation of the two replicas. If a top-level node matches, then Dynamo doesn't descend that branch. If a node in the tree don't match, they copy the branch of the new version to the old one. The use of Merkle trees allows the authors to copy only the parts of the tree that are different. Wikipedia has a picture to illustrate: https://en.wikipedia.org/wiki/Merkle_tree.

Some deeper questions

Considerations/implications for database design?

Is symmetry (lack of leader) important?
Is it really important for every node to be able to service a write?
Can you achieve low latency and high write availability with just one RW replica?

Acknowledgements

Thanks to Zongran, Bernard, Thomas, Gary, and Melodies for asking excellent questions and for answering them when I could not :P

Bibliography

MIT's 6.824 Distributed Systems, Lab 2: Raft

2021-03-16T00:00:00Z

Introduction

Went out on

See my notes on the paper here

Lab 2A: Leader election (moderate)

Finished on Sunday 21st Feb 2021.

Moderate difficulty.

Lab 2B: log (hard)

Started Saturday 27th Feb 2021. Went to NUS and worked from 3pm to 3am with a lot of breaks in between.

Log is 1-indexed: add a dummy element inside log

28th Feb 2021: 12:30pm: Passed TestBasicAgree2B and TestRPCBytes2B, failed the rest.

Tuesday 2nd March 2021: Consistently passes all tests. 2B is now complete.

Some of the bugs I inadvertently introduced:

In TestFailAgree2B one of the servers is disconnected. Since it is disconnected it becomes a candidate and starts to increment its own term again and again. The other two servers happily stay at term 1 and replicate log entries. When it is reconnected, it rejects the AppendEntries because its term is higher. Then the leader immediately becomes a follower, and leader election should occur. Since the disconnected server does not have any of its logs

This is all correct behaviour so far, but for

In my helper function logAtLeastUpToDate I wrote < when I meant to write >. This mea

So the issue is that if let's say you have a node 1 that is a follower and has randomly initialised with some high election timeout, and has some logs, and you have a node 2 that has not up-to-date logs but has initialised with a low election timeout. You can have a situation where node 1 keeps rejecting node 2, but node 2 keeps refreshing / incrementing term and randomly initialising state, and because node 1's election timeout is quite high, node 2 will keep asking, and we'll be unlikely to reach a consensus. This isn't a deterministic bug, it only happens sometimes, we need node1 to have a high election timeout

Solution: Don't reset lastHeardFrom if I receive a RequestVote entry and I do not grant a vote: only refresh lastHeardFrom if i) I grant a vote to a candidate in a RequestVote RPC call, or ii) I receive an AppendEntries RPC call with the same or greater term.

Lab 2C: persistence (hard)

Before I implement Lab 3

Lab 2D: log compaction (hard)

Memcached at Facebook (2013)

2021-03-15T00:00:00Z

Introduction

I read the following the paper as part of my self-study of 6.824 with NUS SoC folks. It may or may not be part of my goal to finish a Stanford CS degree in a year.

This week I watched the 6.824 lecture video Lecture 16: Cache Consistency: Memcached at Facebook. and we discussed the paper Nishtala et al. (2013). Scaling Memcache at Facebook.

Memcached is an in-memory hash table providing storage at low cost.

The big picture

The idea is to reduce the load on the database layer by caching reads. For FB's usecase they have orders of magnitude more reads than writes, and so caching can have significant advantages.

Additionally, it's OK that reads don't necessarily observe the latest writes. So consistency (the condition that all reads observe the latest write) can be relaxed. e.g. a write can be an edit to a post, or a new post. It's OK if you refresh the page a few seconds after a new post has been made and don't see anything. What we want to avoid is unbounded inconsistency: after some time period $t$ we want to guarantee that all users see the latest version of the data.

Second, read operations fetch data from a variety of sources, e.g. MySQL, HDFS, backend services: this points to a separate storage layer that can handle heterogeneous

Memcached provides set, get, and delete.

Memcache is a "demand-filled look aside cache." in contrast to an inline, or "look-through" cache.

(Look-aside means you're aware of the cache, you decide which data paths you want to cache. Look-through means you can't choose to cache or not.)

Get: retrieve from cache: if miss, get from DB. Set: update DB, then invalidate the cache.

Generic cache: use it as a general key-value store, storing pre-computed results from ML algorithms.

As is, memcached runs only on a single server: FB scaled it out to run on a distributed system.

Two key design goals:

any change must impact a user-facing or operational issue: optimisations not considered.
The probability of reading transient stale data as a parameter to be tuned, similar to responsiveness.

Web servers must communicate with many memcached servers, all web servers communicate with all memcached server in a short period.

This all-to-all communication pattern can cause incast congestion or allow a single server to become the bottleneck for many web servers. Data replication alleviates the single-server bottleneck but leads to significant memory inefficiencies in the common case.

Guarantees

Memcached at Facebook promises "best-effort" eventual consistency. Without the cold cluster warm up (more on this later), Memcached guarantees eventual consistency in the absence of network partitions, and loses consistency in the presence of network partitions.

Eventual consistency means that after some bounded time, all reads will eventually return the latest write.

Memcached does not guarantee eventual consistency due to the cold Cold cluster: The two-second holdoff rule fails eventual consistency and results instead in "best-effort" eventual consistency: if the DB hasn't deleted the cache entry in the warm cluster within 2 seconds, then the cold cluster will read a stale value from the warm cluster and that value will be indefinitely inconsistent.

It also doesn't guarantee eventual consistency in the presence of network partitions, because a put request can fail/be partitioned after updating the DB but before invalidating the cache. Then that value will again be indefinitely inconsistent.

Glossary

fan-out: one web server needs to talk to lots of memcache servers to get the data they need.
consistent hashing: reduces the need for rehashing when the number of buckets grows or shrinks
incast congestion: problem at the web server side. When all the responses from the memcache layer are coming back, they get stuck.
single server becoming the bottleneck: when you have a hot key.
load-link/store-conditional:
hold-off-time: Hold of period of two seconds. If a miss is detected in the cold-cluster C, the It can get the stale value, but it won't store the stale value in C.
system V shared memory: just generic shared memory

Three techniques to reduce latency

How to reduce latency? Parallelise and batch, connection coalescing, and sliding window. They build a DAG representing the dependencies, then batch them.

Each server has a library

Standalone proxy (web server) mcrouter: presents a memcached server interface and routes requests/replies to/from other servers.

--> why would u use a proxy.

Clients use UDP and TCP, favouring UDP for get requests to reduce latency and overhead.

(UDP use memcache to do amplification attack)

UDP might detect packets that are dropped or received out of order, we just call it a cache miss.

For set and delete, we need reliability. So we need TCP. Since TCP are stateful, each of these threads must have a separate connection. We use mcrouterto coalesce connections within a machine/server, so instead of in a per-thread basis it's per-machine basis.

Preventing incast congestion

Each server will send multiple requests to different memcache servers, and this may overload the server when the memcache responses all hit the server at the same time. To prevent this, we limit the number of open connections.

Strike a balance where you have too small window size, vs incast congestion with too large a window size.

Reducing Load

Leases address two problems: stale sets and thundering herds. A stale set

Thundering herd: a specific key undergoes heavy read and write activity, which repeatedly invalidates the recently set values, causing all of them to default to the more costly path.

What's a lease: when you as client 1 get a cache miss, the memcached server gives you a 64-bit token. This 64-bit token is a lease: it allows you and you only to

There must be a timeout for this lease Each token lasts only for 10 seconds: subseuquent

Requests for a key's value within 10 seconds of a token being issued results in o

So if the client crashes, it will issue a new token after ten seconds to the next client who sends in a read request.

... explain this section better;

Using lease to minimise application wait time.

We can further reduce this wait time (10s) by identifying situations in which returning slightly out-of-date data is acceptable. When a key is deleted we put it in a temporary store, and if clients are happy to receive stale data, we can give them this recently deleted data.

Memcache pools

There's different types of data: some data is frequently accessed but cache misses are inexpensive, and

Many high churn items will eventually evict low churn items, so we separate different pools.

Replication between pools

storing multiple copies of the same data; they only replicate when the data set is small, requests ask for many keys simultaneously, and the request rate is much higher. They can either do replication or do key-space partitioning. Given the fact that the difference in memcached overhead for retrievin With partitioning

Handling failures

Gutter servers are spun up when memcache servers fail

Clients don't invalidate keys in a gutter server

You hit the DB once and store it in the gutter server. Gutter server prevents lots of DB hits.

How do we deal with multiple regions?

Web server writes to the database directly.

Each webserver has one MCRouter, each CLUSTER also has multiple additional MCRouters

Since cross-cluster communication is expensive, we want to reduce this as far as possible. Hence the cluster-wide mcrouter.

Regional pool

We store cold keys in a shared memcached

Cold cluster warm up

We let cold clusters read from the warm cluster, but this introduces the possibility of inconsistency.

What's the difference between look-aside vs look-through cache?

Look-aside means you're aware of the cache, you decide which data paths you want to cache. Look-through means you can't choose to cache or not.

What Facebook needs

You don't need linearisability, but you cannot cache stale data indefinitely.

Also, user cannot see stale data if they update their own data.

Key concern: we want the user who writes the data not to see their stale data.

Also, we must never expose the database layer to the full brunt of the read requests.

READ

v = get(k). Client hashes the data into k.

# READ 
# where get_cache and set_cache are RPC calls to the Memcached layer
v = get_cache(k)
if v is None:
    v = db_fetch(k)
    set_cache(k, v)
    
    
# WRITE(k, v)
send (k, v) to DB
delete_cache(k)

When FE sends a write to the DB, it will send a DELETE to the rest of the replicated DB

Invalidate scheme vs update scheme?

What's the difference? Why does FB use invalidate instead of delete?

(37:00)

Partition vs replication?

(38:00)

Partition is RAM efficient but not good for hot keys. Clients need to talk to every partition. Replication: good if hot keys, few TCP connections (each client only needs to talk to one partition), but less total data stored.

Why do you need the delete done by the client?

The front-end needs to be the one to do it to ensure that writes by the user are not stale. (The user observes non-stale data).

Bibliography

MIT 6.824 Memcached lecture notes MIT 6.824 Memcached FAQ

Appendix

Why Memcached at FB is not consistent

First let's talk about the key invariant of any database. A database must give the following guarantees (invariants):

Once something is written, it must be readable;
All reads must observe the latest writes.

We say a system is consistent if for every valid concurrent history (a history of interleaving concurrent events that could happen in normal operation) there exists some rearrangement under the consistency rules that preserves the invariants of the database.

So let's consider a particular concurrent history that is problematic. Any write needs to do two separate steps: update the DB, then invalidate the cache entry. But suppose there's another read request that comes in before you can invalidate the cache entry. Then you'll end up serving a stale value to the read.

write1 db start
write1 db end
write1 cache start
write1 cache end
read start
read end (now the cache is filled up)
write2 db start
write2 db end
read start
read end
write2 cache start
write2 cache end

Here the last read does not observe the latest DB write because the second write has yet to delete the cached entry by the time the last read reads from the cache. This breaks the invariant.

Now the challenge is to rearrange this history by the rules allowed by various concurrency models

This history will result in a stale read, thus breaking the invariant. This is obviously not linearisable, since there's nothing to rearrange without changing the real-time ordering.

Might the system be serialisable? I think it is, pathologically. A serialisable database can always reorder all the reads in front and put all the writes after all the reads, so all reads return 0. This is an incredibly weak condition for a system that only allows reads and writes as atomic transactions! See Fauna blog post on why serialisability is useless:

But let's assume that we don't allow that. Let's use strict serialisability. But then of course it isn't strict serialisable for the same reason it isn't linearisable.

Notes on Amazon Aurora (2017, 2018)

2021-03-07T00:00:00Z

Introduction

I read the following two papers as part of my self-study of 6.824 with NUS SoC folks. It may or may not be part of my goal to finish a Stanford CS degree in a year.

The two papers I read are

While the papers were great, they were not detailed enough for my taste. They kind of gloss over the details of how the system is implemented, and they don't really give a step-by-step breakdown of how reads and writes are done and what state is being kept/sent/propagated.

So here I want to first give a big-picture view: what Aurora does and why it's important, then a step-by-step breakdown of how writes, reads, and commits are performed in Amazon Aurora.

Here I assume that you have already read the Aurora paper and I'm not going to mention things that can be easily looked up in the paper. I will instead try to explain what I believe to be their motivations for building this system, the key contributions of the papers and elaborate on some things I believe are underspecified.

Glossary

AZ: availability zone
OLTP: Online transaction processing
PG: protection group. An Aurora storage service is divided into six protection groups that replicate the data six-way.
MVCC: Multi-view concurrency control. This ensures that read replicas always read the freshest possible version of the databases.
MTR: Mini-transaction. Each database transaction in MySQL is a sequence of ordered mini-transaction (MTRs) that are performed atomically. A mini-transaction is made up of multiple log entries.
VCL: Volume Complete LSN. This is the highest LSN on the storage node for which all previous logs are available.
CPL: Consistency Point LSNs. This is a point where it's safe to commit, and the point that will be reached after recovering from a crash.
VDL: Volume Durable LSN. This is the highest CPL that is smaller than the VCL.

The big picture

Amazon is trying to make its normal SQL database more fault tolerant. The easiest way to make databases fault tolerant is to have three replicated databases, and have a quorum of 2 for all writes/reads. But this isn't good enough for Amazon's purposes: Amazon wants fault tolerance at the level of one availability zone (AZ) failure + one additional node failure. (An availability zone is a geographic region: the idea is that if say the entire US blacks out, you can use the Asia and South America replicas.) With three nodes in different AZs, you can't reach quorum if two nodes fail. Hence we need to use six replicas.

But if we use six replicas, the problem that results is that i) databases are chatty and ii) write requests in traditional DB systems are synchronous and the bottleneck quickly becomes network I/O. In a traditional database system this would mean large latencies in write requests.

So the key question Aurora aims to answer is: how can we get greater fault-tolerance while keeping read/write latencies low? Aurora aims to reduce these latencies while giving us the guarantees of a traditional relational DB.

What Aurora is and what makes it special

Amazon Aurora is a CP system. It gives you the guarantees of a relational DB, while being much quicker. But it's not available, because if there's a network partition, it can't reach quorum (more on this later). This is different from Amazon's DynamoDB, which is AP and eventually consistent.

How does Aurora do it? In my opinion, Aurora makes two key contributions: separating the database layer from the storage layer, and making the log the database. Aurora separates the database layer from the storage layer, which allows writes to be done asynchronously without having to wait for the data to be replicated to disk. The database layer applies writes to its own buffer cache after it receives confirmation from the storage layer. This allows the database to service client requests (with its cache) while previous write requests are written and replicated asynchronously on the storage layer. Pages are (again, asynchronously) generated by the storage layer and surfaced when needed in response to a read request. This solves the latency issue.

Another way they reduce latency is by decreasing the amount of data that needs to be replicated by sending only the write-ahead logs: "the log is the database." They use a strictly increasing log number called the Log Sequence Number (LSN) to make sure logs are consistently replicated across the six replicas.

In more detail

There's one primary read-write instance (leader) and multiple replica read-only database instances. There's only one storage volume, replicated six ways (each replica is called a PG, or "protection group") in three different AZsf.

Write requests

What exactly happens when Aurora needs to service a write request?

The primary replica is the main source of truth. It keeps some important state. The most important number is the Log Sequence Number (LSN). It also keeps volume durable log (VDL) records and current epoch number.
The primary replica receives a write request, which is a database transaction.
The database-level transaction is broken up into multiple mini-transactions that are ordered and must be performed automically.
Each mini-transaction is itself composed of multiple contiguous log records. Because the primary replica is the main source of truth, it knows exactly what LSNs it needs to issue for each log record: this is important because of failure recovery and truncation range.
These log records are sent asynchronously to the storage service. Each storage node persists the log records to an update queue, and responds with an acknowledgement, along with the Segment Complete LSN (SCL). All of this is asynchronous.
When the primary replica receives a quorum acknowledgement from 4 out of 6 storage nodes, that write is considered done. When the replica observes SCL advance at four of six members, it locally advances the "Protection Group Complete LSN" (PGCL), which is the point at which the protection group has made all writes durable.
If an MTR is complete (the quorum SCL > the LSN of the last log entry in the MTR), The primary instance may ship the MTR chunks and the new VDL to its read replicas, which applies them sequentially to generate its own cached version of the pages.

The database instance acquires latches for each data block, allocates a batch of contiguously ordered LSNs,generates the log records, issues a write, shards then into writebuffers for each protection group associated with the blocks, andwrites them to the various storage nodes for the segments in theprotection group. We use an additional consistency point, the Vol-ume Durable LSN (VDL), to represent the last LSN below VCLrepresenting an MTR completion.

Replicas do not have the benefit of the latching used at thewriter instance to prevent read requests from seeing non-atomic structural updates. To create equivalent ordering, we ensure that log records are only shipped from the writer instance in MTR chunks. At the replica, they must be applied in LSN order, applied only if above the VDL in the writer as seen in the replica, and applied atomically in MTR chunks to the subset of blocks in the cache. Read requests are made relative to VDL points to avoid seeing structurally inconsistent data

Read requests

Aurora cleverly avoids quorum reads (that is, querying all six PGs, and waiting for the quorum of 3). Why would you ever need a quorum read? If you don't do a quorum read you run the risk of querying a node that has stale data: e.g. you read from a node that's one of the two nodes that wasn't included in the write quorum. The primary replica never needs to do a quorum read, obviously, because it knows exactly which PGs have the latest writes.

But how do the read replicas avoid quorum reads? The read replicas are sent the VDL by the primary replica, and I'm guessing if they query a node whose CPL (Consistency Point LSN) is lower than the VDL (the VDL is the highest CPL that is smaller than equal to VCL), they retry with another node?

When reading a page from disk, the database establishes a read-point, representing a VDL at the time the request was issued. The database can then select a storage node that is complete with respect to the read point, knowing that it will therefore receive an up to date version

Commit requests

Recovery

Epoch numbers

The idea of "epochs" are very similar to the term value in Raft. Epochs are used to invalidate stale requests after a crash in the database instance.

How a database works

Traditional relational databases organize data in pages, and as pages are modified, they must be periodically flushed to disk. For resilience against failures and maintenance of ACID semantics, page modifications are also recorded in redo log records, which are written to disk in a continuous stream. While this architecture provides the basic functionality needed to support a relational database management system, it's rife with inefficiencies. For example, a single logical database write turns into multiple (up to five) physical disk writes, resulting in performance problems.

I don't really understand how databases work, and why they need to write so many things.

Thankfully the 6.824 lecture notes on Aurora has a very helpful explanation:

Paper assumes you know how DB works, how it uses storage.
here's an overview of a generic transactional DB
single machine
example bank transaction -- transfer $10 from x's account to y's account
x = x + 10
y = y - 10
locks x and y, released after commit finishes
data in b-tree on disk
cached data pages
DB server modifies only cached data pages as transaction runs
and appends update info to WAL (Write-Ahead Log)
the paper's "redo log" == MySQL's INNODB transaction log.
log entries:
LSID  TID  key  old  new
101     7    x  500  510     (generated by x = x + 10)
102     7    y  750  740     (generated by y = y - 10)
103     7  commit            (transaction finished)
commit once WAL safe on disk
release locks on x and y
reply to client
later write modified data pages from cache to disk
to free log space, allow cache eviction
crucial to performance to delay -- write absorbtion, since pages are big
crash recovery
replay "new" values for all committed xactions in log (redo)
replay "old" values for uncommitted xactions in log (undo)

Traditional databases have to write so many things over the network because there is no compute in your storage: the storage is dumb. It can't generate the pages from the redo log.

What is a write-ahead log?

From Wikipedia:

In computer science, write-ahead logging (WAL) is a family of techniques for providing atomicity and durability (two of the ACID properties) in database systems. The changes are first recorded in the log, which must be written to stable storage, before the changes are written to the database.

In Aurora the write replica sends a write request to the storage service, which writes it to an update queue (this is the write ahead log!). Only this step is synchronous, and responsible for the write latency.

But once it's written to the four update queues (you need four nodes for quorum), the database service can be assured that this write will succeed, and can return the client a result. The results can then be slowly persisted to disk asynchronously by the storage service, freeing up the database service to service more requests.

What's a data page?

Databases store data in terms of pages, pages tend to be 4KB. In general, each row is in a page (barring very large rows) and is written to disk.

If the Aurora DB doesn't write a lot of this stuff to disk, why do these old DBs need to write to disk? What is the tradeoff that we're making here?

The key tradeoff is that your storage now needs compute. Your storage has to do the job of taking the log and building pages from it, where previously this was done by the database instance.

FAQ

Why might read replicas get stale reads?

The redo log seen by a read replica does not carry the stateneeded to establish SCL, PGCL, VCL, or VDL consistency points.Nor is the read replica in the communication path between thewriter and storage nodes to establish this state on its own. Notethat VDL advances based on acknowledgements from storage nodes,not redo issuance from the writer. The writer instance sends VDL update control records as part of its replication stream. Althoughthe active transaction list can be reconstructed at the replica us-ing redo records and VDL advancement, for efficiency reasons weship commit notifications and maintain transaction commit history.Read views at the replica are built based on these VDL points andtransaction commit history

VDL on the replica may lag the writer -- hence read replicas may get stale reads. How? This may happen if the primary instance has already written and committed and advanced the VDL locally but hasn't sent "VDL update control records" yet as part of its replication stream to the secondary replica.

Why don't database instances have to flush dirty pages?

What is the yellow arrow in Figure 2 point to Amazon S3? What is a binary (statement) log?

[TODO] not sure..

How does the storage service scale out I/Os in an "embarassingly parallel fashion"?

Firstly, the storage nodes are divided into independent storage segments of 10GB each, this means that nodes can spin up and down very quickly and segments can be repaired in 10 seconds over a 10 Gbps connection. So this scales very well. What I think they mean by "embarassingly parallel" is that each storage node (or PG, I suppose) is independent, and there's no additional bookkeeping to keep each node in sync. Hence the database isntance can send to all these PGs simultaneously and all of them can process the writes in parallel.

Why don't you need a quorum read? How does Aurora ensure that it gets the latest version of the read without a quorum read?

See the top section on this

What is a read replica?

Section 3.2 of the 2018 paper ("Scaling Reads using Read Replicas"):

Many database systems scale reads by replicating updates froma writer instance to a set of read replica instances. Typically, thisinvolves transporting either logical statement updates or physicalredo log records from the writer to the readers. Replication is donesynchronously if the replicas are intended as failover targets with-out data loss and asynchronously if replica lag or data loss duringfailover is acceptable

Aurora supports logical replication to communicate with non-Aurora systems and in cases where the application does not wantphysical consistency – for example, when schemas differ. Internally,within an Aurora cluster, we use physical replication. Aurora readreplicas attach to the same storage volume as the writer instance.They receive a physical redo log stream from the writer instanceand use this to update only data blocks present in their local caches.Redo records for uncached blocks can be discarded, as they can beread from the shared storage volume

This approach allows Aurora customers to quickly set up and teardown replicas in response to sharp demand spikes, since durablestate is shared. Adding replicas does not change availability ordurability characteristics, since durable state is independent fromthe number of instances accessing that state. There is little latencyadded to the write path on the writer instance since replicationis asynchronous. Since we only update cached data blocks on thereplicas, most resources on the replica remain available for read re-quests. And most importantly, if a commit has been marked durableand acknowledged to the client, there is no data loss when a replica

What happens when a storage node fails after acknowledging the primary instance (e.g. written to Update Queue), but before it persists that data to disk?

This doesn't happen, because the update queue is persisted to disk.

What's the difference between a write and a commit?

A write is a single write operation, while a commit is committing an entire transaction, which can contain multiple write and read transactions. Committing a transaction means making it atomic: saying that ALL the operations in that transaction have
passed (or failed).

What are the differences between Aurora and Raft?

Obviously, Raft is a consensus algorithm, while Aurora is a database deployment. It's more accurate to ask the question: how does Aurora manage to maintain a consistent state and how does it compare to the way that Raft does it?

At first glance they seem quite similar. There is one leader which is the source of truth. This leader writes to a log, which is then applied to get the state. Aurora's "epoch" is a monotonically increasing value, much like the "term" in Raft: it's provided as part of every read or write to a storage node, and storage nodes will not accept requests at stale volume epochs.

The key assumption Aurora can make is that there is a common shared state. In Raft we can't assume that we have e.g. the provably correct term number, the provably correct leader, even the provably correct log. and what not. So there's a difficult problem of consensus; we need to agree on a leader, agree on the term, without a single source of truth. In Aurora there's no need for consensus: you literally just read the log from the storage service, and read the VDL/epoch/etc. directly from the metadata store.

In this sense, Aurora isn't really a distributed system at all! We always know the VDL, largest LSN, and epoch, because these are all persisted in the metadata store. And once we have quorum from the four out of six storage nodes, we're guaranteed that we will always know the correct VCL. The only distributed bit is the storage service, but again there's no issue with consensus because each storage service HAS to append its logs as long as the epoch number is valid. The storage service has to query the storage metadata service before servicing any read or write request (page 792 of 2018 paper). (what happens when the storage service receives a stale volume epoch though?)

What are the similarities and differences between Aurora and Memcache?

Do replicas apply log records ABOVE or BELOW the VDL?

2017 paper section 4.2.4, "Replicas" says:

The replica obeys the following two important rules while applying log records: (a) the only log records that will be applied are those whose LSN is less than or equal to the VDL, and (b) the log records that are part of a single mini-transaction are applied atomically in the replica's cache to ensure that the replica sees a consistent view of all database objects.

2018 paper section 3.3, "Structural Consistency in Aurora Replicas" says:

To create equivalent ordering, we ensure that log records are only shipped from the writer instance in MTR chunks. At the replica, they must be applied in LSN order, applied only if above the VDL in the writer as seen in the replica, and applied atomically in MTR chunks to the subset of blocks in the cache. Read requests are made relative to VDL points to avoid seeing structurally inconsistent data.

So is it "less than or equal to", or "above" the VDL??

EDIT: Thanks Zongran:

So the 2017 statement means that you only apply the redo logs whose LSN is below VDL as they are considered durable by quorum; 2018’s statement has this context that the read instance’s view of the write’s VDL is lagging, and hence you only need to make up using those redo logs from write instance whose LSN is higher than your current view of the writer’s VDL

Bibliography

https://blog.acolyer.org/2019/03/25/amazon-aurora-design-considerations-for-high-throughput-cloud-native-relational-databases/
https://www.allthingsdistributed.com/2019/03/Amazon-Aurora-design-cloud-native-relational-database.html
MIT Lecture Notes on Aurora
MIT Aurora FAQ

FLP Impossibility Theorem

2021-03-01T00:00:00Z

What does asychronous mean?

You have a distributed system

Introduction

We work under message passing model Asynchronous: clocks are unreliable, non bounded clock skew no assumptions on timing

Question: N nodes have a value between 0 and 1. Eventually all nodes must agree on a value that is meaningful (non-vacuous). If all nodes were given 0, then we have to agree on 0, and if all nodes were given 1, then we have to agree on 1.

Given a set of processors, each with an initial value:

All non-faulty processes eventually decide on a value

All processes that decide do so on the same value

The value that has been decided must have proposed by some process

These three properties are referred to as termination, agreement and validity. Any algorithm that has these three properties can be said to solve the consensus problem.

What is the relation with the two generals problem?

The FLP is a generalisation of the two generals problem, for N.
If the two generals have a shared clock, this is possible.

Impossibility theorem:

Also your messages will always be delivered. I may crash any one of you guys.

From your perspec

Processors are allowed to fail according to the fail-stop model - this simply means that processors that fail do so by ceasing to work correctly. There are more general failure models, such as Byzantine failures where processors fail by deviating arbitrarily from the algorithm they are executing. Again, by strengthening the model - by making strong assumptions about what is possible - the authors make their result much more general. The argument is as follows: if a given problem is proven unsolvable in a very restricted environment where only a few things can go wrong, it will certainly be unsolvable in an environment where yet more things can go wrong.

Any such algorithm that tries to give an answer in bounded time must sometimes be wrong.

Statement of the theorem

There is no algorithm that solves the problem ==> any algorithm that terminates in finite time is

Terminology

We have a vector of states $S_1, ... S_n$ We allow nodes to send or receive messages: the conent of the message, and the recipient of the message. $(p, m)$ where $p$ is the recipient and $m$ is the message.

Messages are always delivered in finite time.

Univalent and bivalent: 0-valent, 1-valent.

0-valent means

Bi-valent: if you have a state $M$ ,

Processes can do one of two things: they can

Allowed operations:

At every step: you may either take a message from the message pile and apply it to the state list, or you may throw a message into the message pile.

Claim: There is no algorithm that starts off in a bivalent state and ends up in a univalent state.

Assume there is a god algorithm that starts with any state list and gives you some output. Ergo, every state you feed it is a univalent state.

Gray Coding: one where you lay out the sequence of n bits and every step you go down you only change one bit. Between any two rows or wrap over back around, you only flip one bit.

There must be two rows that differ in exactly one bit, and have different values 0 and 1 (Proof is trivial.) But if you have to tolerate one crash, then

=> Ergo, there must be some state that is bivalent for any algorithm

And the idea is that we feed in a bivalent state, and keep it bivalent while passing the messages.

we have a message $e = (p, m)$ in the message pile, and there's a way to keep it bivalent

Let C denote the set of states that can be reached from T without delivering this message.

And let D as the set of states that can be reached from T with delivering this message.

In order to keep the entire system bivalent, there must be one bivalent state in D.

Assume by contradiction. So every state in D must be univalent

If a configuration in C is zero valent, delivering a message in C0 will result in a state that is also zero valent. Ditto with one-valent.

If $p$ and $p^{'}$ are different (different receivers), we can reorder the message order, ... F0 to F1 because there will be

It cannot be the case that they are different.

Notes on technical mentorship

2021-02-21T00:00:00Z

Introduction

I thought Richard Artoul's post From coding bootcamp graduate to building distributed databases on HN today was excellent.

What can I learn from it and how can I maximise my own technical mentorship?

Take responsibility for your career trajectory

At some point in your career you’ll have to start taking responsibility for your career trajectory. The key insight is that you should make decisions about what projects to work on and which teams and companies to join strategically, not tactically. Think long term.

In the grand scheme of things, a 10% to 30% difference in compensation for a few years is small in the grand scheme of your career. After my first two years at Uber I could have switched companies for an easy 30% to 40% boost in my compensation, but instead I made a strategic decision to position myself on a team where I was paid “below market”, but where I was developing skills that I would leverage to achieve a 3x jump in my compensation later.

In particular, there are two areas that you should pay particular attention to:

Finding good mentors
Developing a valuable and unique skill set

Do whatever it takes to find good mentors

Nothing will accelerate your growth faster than spending all day working with other very good engineers. The moment you start to feel like you’re not learning from the engineers around you is the moment you should start looking for a new team.

Keep in mind that when I use the term “mentor” I mean engineers that you work with everyday, participate in design discussions with, and who review your code on a regular basis. This is very different from a “mentor” that you solicit career advice from once a month over coffee.

There is no exact formula for finding good mentors. The key is having the humility to recognize those people when you meet them, and to be willing to make sacrifices to be around them.

If you work at a large company, this process shouldn’t be too difficult. Start asking around and try to identify a high performing team with very good engineers doing impactful work, and then do whatever you can to start working with them. Offer to help out on their projects on your own time, or if you’ve been at the company long enough and have built up some social capital, try and transfer to that team. Your manager may not be happy about it, but hiring is hard and the manager of the team that you want to join will likely extend the necessary political cover to make the transfer happen if they think they can get a coachable and motivated junior engineer with a proven track record “for cheap”.

In my case, I was working at Uber and had been working on the same team for 18 months and was stagnating technically. I told my manager that I was going to start looking for a new team and then I spoke with the managers of 15 different teams at the company. The team that seemed to have the most skilled engineers doing the most technically interesting and impactful work was located in New York, a city I had no interest in moving to, but I flew to NY on my own dime and spent 3 days working with their team on a small task. After meeting the team and seeing how skilled they were, I knew I had to move to New York even though I didn’t know anyone there. In hindsight, that decision completely altered the trajectory of my career and I have no doubt that I wouldn’t be where I am today if I hadn’t prioritized technical mentorship over everything else that was going on in my life at the time.

Obviously dropping everything and moving across the country isn’t an option for everyone, but if your goal is to get dramatically better then finding mentors should be one of your top priorities.

It’s also worth emphasizing that I interviewed with 15 different teams. That’s a lot of opportunities to investigate! For context, after I graduated from my bootcamp I had 3 onsite interviews before landing at Uber. If you assume that opportunities follow some kind of power law distribution such that some opportunities are significantly more valuable than others, then going through this exercise whenever possible will likely lead to much better outcomes. If you then factor into the equation that these types of decisions compound over the lifetime of your career it starts to seem grossly negligent to not take this exercise seriously everytime you’re gearing up for a major change.

How to find good mentors if I'm not in a large company with very good engineers?

If you work at a large company, this process shouldn’t be too difficult. Start asking around and try to identify a high performing team with very good engineers doing impactful work, and then do whatever you can to start working with them.

In hindsight, that decision completely altered the trajectory of my career and I have no doubt that I wouldn’t be where I am today if I hadn’t prioritized technical mentorship over everything else that was going on in my life at the time.

In his telling, technical mentorship is very important. It's great that he was at Uber. I am not at Uber. What can I do instead?

Here's Atul Gawande on finding a coach for his surgery (got his old professor to observe his surgeries) Personal Best This is bootstrapping technical mentorship, in a way. What can I learn from this?

Some rough notes on React after playing with it for a day

2021-02-19T00:00:00Z

Introduction

I'm preparing for an upcoming interview where I'm expected to know React. As practice I built a very ugly, very silly page that allows you to customise your food to the gram (some names considered and rejected: "Food for Autists", "Macro Madlads") last night in a couple of hours.

lieu, [19.02.21 13:06] A website that allows you to build your own bowl (carbs, protein, veg, etc) but the macros are automatically updated

lieu, [19.02.21 13:06] so you can pick exactly how many grams of chicken breast and pasta you want

lieu, [19.02.21 13:06] Then pay and then we offer islandwide delivery within an hour's time

lieu, [19.02.21 13:06] Carbs: rice (brown/white), pasta (0g is an option) Protein: chicken breast, oven baked salmon, beef steak, tofu, egg Vegetables: random salad vegetable, kimchi, broccoli, edamame, cherry tomatoes. Price: base bowl/delivery cost + all items sold by weight, carbs and protein accurate to the gram.

The idea is not that important: the main objective here is to practice writing clean, idiomatic, modern React code. GH repo here.

How do you pass parameters into a callback function?

Suppose you have the following React components where a child element calls a callback function onChange

function Parent() {
    function handleChange() {
        // do something...
    }
    return <Child handleChange={handleChange}/>
}

function Child() {
    return <input onChange={props.handleChange}/>
}

But what happens if you want to pass some parameters to the parent function? I was stuck on this for quite a while until Nick pointed this out to me. Don't think of passing parameters into a callback function per se; rather, create a higher-order function that then calls the parent's callback function.

function Parent() {
    function handleChange(value) {
        // do something with the value
    }
    return <Child handleChange={handleChange}/>
}

function Child() {
    return <input onChange={(e) => handleChange(e.target.value)}/>
}

[?] Although I wonder if handleChange would be already implicitly called with e as the argument...

Hooks TL;DR

useState: takes in an initial state object, returns a state object and a setState function that mutates that state object.
useEffect: takes in a function that usually performs side effects. That function may return a function. If so, the _returned _function is run when the
useReducer: takes in an initial reducer function and an initial state and returns [state, dispatch]. state is straightforward, just like the one in useState. What's dispatch and how is it different from setState?

Internally, Hooks are implemented as linked lists. When you call useState, we move the pointer to the next item. When we exit the component’s “call tree” frame, we save the resulting list there until the next render.

(source: React as a UI runtime)

What is the `dispatch` method in the `useReducer` hook?

The useReducer section of the Hooks Reference API writes:

useReducer

const [state, dispatch] = useReducer(reducer, initialArg, init);

An alternative to useState. Accepts a reducer of type (state, action) => newState, and returns the current state paired with a dispatch method. (If you’re familiar with Redux, you already know how this works.)

I'm not familiar with Redux, and I don't know how this works. The reducer function is simple enough: it's a function that takes in a state and an action object and returns a new state object. But what exactly is the function signature of dispatch? Well, it turns out that another section of the React docs Building Your Own Hooks has it. The dispatch function simply calls the provided reducer function on the state to get the new state object, nextState. It then mutates the state using setState(nextState).

function useReducer(reducer, initialState) {
  const [state, setState] = useState(initialState);
  
    function dispatch(action) {
        const nextState = reducer(state, action);
            setState(nextState);
              
    }
    
      return [state, dispatch];
      
}

Here's how we would use this dispatch function:

function Todos() {
  const [todos, dispatch] = useReducer(todosReducer, []);
    function handleAddClick(text) {
        dispatch({ type: 'add', text  });
          
    }
      // ...
}

You can think of dispatch as a "higher-level setState": while setState merely replaces one state with another, dispatch allows us to specify exactly how we want to setState according to the reducer function.

What's `useReducer` for? Why use it over `useState`

The difference between useReducer and useState is that useReducer allows you to pass it a reducer function. useReducer also returns a dispatch function that is the equivalent of setState.

Why would you use dispatch? Why not just pass the setState function down as props directly? You can certainly do that, but there are two advantages to using dispatch.

Firstly, using dispatch allows us to simplify complex updating functions, as well as encapsulate all of the logic in one single reducer function. If you didn't have that function each child component would have to implement their own function to create a new state before setting it. With the dispatch pattern you can simply pass it some parameters and the reducer function will handle it. [TODO] (... i'm not entirely convinced. come up with an example where it's obvious that the dispatch method is significantly better than just passing setState down)

Speaking of passing props: You can use dispatch with useContext to avoid passing props many levels down. It's a way for deeply nested components to mutate the state of the parent which might be many levels up. I remember first writing React in 2015 and thinking that it was very very leceh to have to pass props down. [TODO] (but again it's unclear why I can't pass the setState function down with context) But what is useContext?

What is context?

[TODO]

React's Context API is a mechanism for making a single user-provided value available to a subtree of components. Any component inside of a given <MyContext.Provider> can read the value from that context instance, without having to explicitly pass that value as a prop through every intervening component.

Bibliography

Splitting a string into the fewest number of palindromes

2021-02-18T00:00:00Z

Started 645pm

Question Number 181

Good morning! Here's your coding interview problem for today.

This problem was asked by Google.

Given a string, split it into as few strings as possible such that each string is a palindrome.

For example, given the input string racecarannakayak, return ["racecar", "anna", "kayak"].

Given the input string abc, return ["a", "b", "c"].

Problem analysis

Call a partition $P$ a list of indices $[i_1, i_2, ... i_K]$ where each index $i_k$ denotes where to split the string. A partition is valid if each substring

S[0:i_1], S[i_1:i_2], ... ,S[i_{K}:]

is a palindrome.

A partition $P$ is minimal if len(indices) is the smallest for all possible valid partitions.

The recurrence relation is as follows: The minimal partition of $S[0:N]$ is the minimum of minimal partitions of all $S[k:N+1] \forall k <= N$ where $S[0:k]$ is a palindrome.

That is to say, for each k where $S[0:k]$ is a palindrome, recurse into the subproblem of finding a minimal partition for $S[k:N+1]$ .

I believe the time complexity is $O(N^2)$ . In the worst case each substring is a palindrome, so you have to check all the substrings. With memoisation each lookup of a substring is $O(1)$ , and checking whether a substring is a palindrome should also take constant time if a subsubstring already exists (checking if $S[a:b]$ is a palindrome is $O(1)$ if you have memoised $S[a+1, b-1]$ ). $O(N^2)$ memory complexity.

Pseudocode:

# TODO

Saltzer and Kaashoek (2009). Principles of Computer System Design: An Introduction

2021-02-15T00:00:00Z

Introduction

I read this paper as part of my self-study of 6.824 with NUS SoC folks. It may or may not be part of my goal to finish a Stanford CS degree in a year.

This week we read the Saltzer and Kaashoek (2009) textbook, which explained atomicity, consistency, locking, two-phase commits, and distributed two-phase commits. Song Tianyi was in charge of explaining this week (click the link for his notes).

I've marked things I don't know with [?] and bits that still need work with [TODO]. Please help me clarify or fill them in!

9.1.5 Before-or-After Atomicity

What is sequence coordination?

[?]

not sure about this

Sequence coordination ensures some logical order/ semantics of the object, e.g. you cannot dequeue before enqueuing.

What is before-or-after atomicity?

Concurrent actions possess before-or-after atomicity if their effect from the point of view of the invokers is the same as if the actions occurred either completely before or completely after one another.

What's the difference between sequence coordination and before-or-after atomicity?

[?]

What's the relationship between before-or-after atomicity and serialisability? What's the difference?

A history H is serialisable if and only if a concurrent history can be rearranged into some (any) serial ordering. "Informally, serializability means that transactions appear to have occurred in some total order."

Before-or-after atomicity implies serialisability. Consider any set of concurrent actions. By before-or-after atomicity (BAA) they must appear to have occurred either completely before or completely after one another. But that directly implies that there exists some serial ordering of these concurrent transactions, which is exactly what we need for serialisability.

What's the difference between consistency models and these atomicity guarantees?

[?]

There is another atomicity guarantee called all-or_nothing atomicity. You need before-or-after and all-or-nothing atomicity in order to get serialisability?

I think before-or-after atomicity implies all-or-nothing atomicity. This is because if all-or-nothing atomicity was violated, it would be possible for half of a transaction to fail and half of a transaction to succeed... so what? is that OK as long as both halves lie before or after other transactions?

What is the fundamental definition of atomicity?

Both atomicity guarantees can be thought of as what we really care about:

An action is atomic if there is no way for a higher layer to discover the internal structure of its implementation.

9.1.6 Correctness and serialisation

The textbook gives this general concept of correctness:

if any result is guaranteed to be one that could have been obtained by some purely serial application of those same actions.

This is serialisability!

Before-or-after atomicity leads directly to this concept of correctness

If all transactions are before-or-after atomic, then they must form some purely serial ordering. And if they form some purely serial ordering, this is the definition of serialisability.

Note that serialisabilty places no constraints on the total order, so transactions may appear to execute in any (serial) order. There are correctness requirements stricter than serialisability: we walk through two here.

There are correctness requirements that are stronger than serialisability: external time consistency, or sequential consistency

External time consistency (that transactions must appear to happen in the order that they were called) is equivalent to strict serialisability. Note that strict serialisability is not quite the same (is stricter than) linearisability:

We can view strict serializability as linearizability plus the multi-object transactions of serializability. But in another sense, linearizability is strict serializability with the constraint that transactions are constrained to act on a single object source

What is sequential consistency?

Sequential consistency is stronger than serialisability. It requires that transactions must appear to have occurred in some order, and that transactions must appear to have occurred in order on a per-process basis.

For example, this concurrent history

B READ x 
A READ x
B WRITE x
A WRITE x

would be serialisable but not sequentially consistent if the following execution order was permitted:

A WRITE x
A READ x
B WRITE x
B READ x

This respects a serial ordering, but is not sequentially consistent because process A needs to read before writing.

What's the difference between external time consistency and sequential consistency?

External time consistency imposes a time order on all different processes. In sequential consistency all transactions within a process must run sequentially. But there's no requirement for real-time order between processes.

9.5.2 Simple locking

We next consider how to maintain correctness. One way is locking. We first consider simple locking. There's a lock manager that manages the locks. Each transaction has to acquire all the locks that it will need before starting the transaction. The transaction has what is called a lock point: the instant at which it has acquired all of its locks. It may release its locks only after the transaction installs its last update and commits or completely restores its data (aborts).

A lock manager can enforce simple locking by requiring that each transaction supply its intended lock set to the function call.

Simple locking can cause deadlocks

[TODO]

Consider the following scenario:

Mitigate this by assigning each lock a unique ID and mandate that each process tries to acquire each lock in order.

Simple locking is not completely efficient

However, simple locking might not be efficient because it needs to:

acquire locks on every shared object that it might need to read or write, even if it doesn't need it in the end;
acquire locks on all objects at the start, even if the transaction is long-running and it may only need the lock towards the end.

So there are some more optimistic approaches. One of them is two-phase locking, which we'll cover next.

9.5.3 Two-phase locking

Two-phase locking improves upon simple locking in two key (ha ha) ways: rather than forcing a transaction to acquire all the locks before it begins, it allows the transaction to acquire locks as it proceeds. The transaction may read or write a data object only after it acquires a lock on that object.

Second, instead of forcing a transaction to release all locks at the end, the transaction may release a lock on an object that it only reads if it will never need to read that object again, even to abort. (Note that it can't release a lock on an object that it writes, because it would be impossible to roll back!)

This is called two-phase locking because the number of locks requested by a transaction increases monotonically at the start, then decreases monotonically.

However, two-phase locking still has its inefficiencies...

[todo]

How can we handle system recovery with locks?

[TODO]

There are two interactions between locks and logs that require thought: 1) individual transactions that abort, and 2) system recovery.

Aborts are easy, since all aborting transactions must restore their changed data objects to their original values before releasing any locks.

Are volatile locks (locks in memory) fault-tolerant?

What happens if you have processes holding different sets of locks and then they crash? How do you make sure that reconstructing the new state from the logs ensures the correct (serialisable) outcome?

Two-phase locking often results in deadlocks

[TODO]

Two-phase locking results in deadlocks but simple locking does not result in deadlocks (why?) as long as you index locks in order. (Prove this -- also mentioned in GFS)

9.6.2 Two-phase commit

[TODO]

Go through this myself

9.6.3 Multiple-Site Atomicity: Distributed Two-Phase commit

[TODO]

In Chapter 4 we learned of the Remote Procedure Call method and in Chapter 7 we learned how to design an RPC with a persistent sender. Now we combine a two-phase commit protocol with persistent senders, duplicate suppression, and single-site transactions to build a distributed two-phase commit protocol.

Correctness of multiple-site atomicity is when all the sites commit OR all the sites abort; we will have failed if some sites commit their part of a multiple-site transaction while others abort their part of that same transaction.

What is the procedure for 2PC?

First phase: Alice the coordinator assigns its child jobs to its slaves: PREPARE TO COMMIT X. Bob, Charlie and Dawn PREPARE to commit X (commit tentatively). They acquire and don't release the locks. (or abort, if they can't): I HAVE PREPARED TO COMMIT X or I HAVE ABORTED X. Second phase: Only after receiving all tentative commits, or any least one abort, will Alice choose to abort the entire transaction or tell a different worker site to carry its component transaction. Alice marks her own outcome record COMMITTED and sends COMMIT back to Bob, Charles and Dawn. Each worker site, upon receiving such a message, changes its state from PREPARED to COMMITTED and exits.

Only three messages are needed: one message to tell the worker to prepare the commit, one message to tell the master that it's prepared, and one message to tell the worker to commit.

Bob, Charlie and Dawn can lose messages arbitrarily, so Alice will never know if they have received her message to commit. However, since Bob, Charlie and Dawn are persistent senders (they keep asking if they haven't received a commit instruction), Alice can be confident that the workers will commit their transactions eventually.

What happens when the coordinator crashes?

[TODO]

What happens when a worker crashes?

There are four places a worker can crash:

Before it receives the first phase message
After it receives the first phase message, but before it sends the prepared message
After it sends the prepared message, but before receiving a commit message
After receiving a commit message

Only state 3 is nontrivial: recovery procedure must reacquire all locks the PREPARED transaction was holding at the time of the failure. The recovery procedure must also restart the persistent sender.

What are the tradeoffs of 2PC?

2PC gives up availability: nodes can block forever if the coordinator goes down.

The Spinnaker paper mentions a few other tradeoffs: Firstly, the failure of a single node will lead to an abort. Secondly, it suffers in performance: a typical implementation of 2PC requires 2 disk forces and 2 message delays. Lastly, it also blocks when the coordinator fails.

Why must the coordinator must hold completion states of all transactions indefinitely?

[TODO]

The Two Generals problem and impossibility theorem

The Two Generals problem is an impossibility theorem that states that it is impossible for two nodes to reach consensus over a lossy network. There's a lovely formalisation of the problem and the impossibility proof here.

Given the Two Generals impossibility theorem, how does 2PC work?

[?] [TODO]

You'd think the two generals problem is very similar to the two-phase commit protocol, so how come 2PC works despite the impossibility result?

Bernard thinks it has to do with the notion of bounded vs unbounded time. I think that's true, but it might be more useful to think of it more as blocking vs non-blocking: we can't decide

There's a really nice blog post on 2PC by Henry Robinson here. I'll have to read this post (and the rest of his blog!) much more closely, but I think here it solves the consensus problem but a node failure might block indefinitely.

Follow-up: what is the relationship between the FLP and the Two Generals Problem?

When would you want to use 2PC?

If you had independent databases with very non-atomic transactions e.g when you buy something, you have child transactions of contacting the credit card company, decrementing the stock, adding loyalty points, etc. And you want to either commit or roll all of this back atomically.

Should one use 2PC or Raft?

They are solving different problems: Raft solves the problem of replication, 2PC solves the problem of coordination. We use 2PC when we want different computers to do different things. We use Raft when we want to replicate the same state over different machines.

What is the relationship between 2PC, 3PC, and Raft?

[TODO]

The major events in the development of consensus protocols can be summarised below, in hugely oversimplified form:

Jim Gray (amongst others) proposes 2PC in the 1970s. Problem: blocks on failure of single node even in synchronous networks.

Dale Skeen (amongst others) shows the need for an extra, third phase to avoid blocking in the early 1980s. 3PC is an obvious corollary, but it is not provably correct, and in fact suffers from incorrectness under certain network conditions.

Leslie Lamport proposes Paxos, which is provably correct in asynchronous networks that eventually become synchronous, does not block if a majority of participants are available (so withstands $n/2$ faults) and has provably minimal message delays in the best case.

(source: the excellent Henry Robinson again)

FOLLOW-UP: be clear about the difference between synchronous and asynchronous networks

FOLLOW-UP: add a point number 4. talking about Raft

Conclusion

[TODO]

Notes on CRDTs

2021-02-15T00:00:00Z

Introduction

This week I read Shapiro et al. (2011). A comprehensive study of Convergent and Conflict-Free Replicated Data Types as part of self-study of 6.824 with NUS SoC folks. It may or may not be part of my goal to finish a Stanford CS degree in a year.

Overview

The big picture

A lot of the literature in distributed systems tries to solve the problem of consistency, which (roughly speaking) demands that all nodes must have the same state at every point in time.

Consensus algorithms like 2PC and Raft ensure consistency by guaranteeing that every transaction is either executed on all nodes, or not at all. This ensures that each node will have the same transactions in the same order. The idea is if every node starts from the same initial state, and each node runs the same set of (deterministic) transactions in the same order, then each node will have the same subsequent states. So this is the approach to solving consistency.

Of course we know from the famous CAP theorem that we give up availability as a result. We can try and get back availability by relaxing consistency into eventual consistency: mandating that all nodes must have the same state at some point in time. This is the approach favoured by Amazon's DynamoDB.

But even relaxing consistency to allow eventual consistency doesn't necessarily give us full availability. The biggest problem is that nodes could reach a "broken" or otherwise unrecoverable state if operations come in out of order. A simple example is a stack: obviously the state of the stack depends on the order of operations, and if the pop operations come before the push operations you're broken.

And even if our nodes never "break", their histories can diverge irreconcilably in a way that kind of breaks eventual consistency --- DynamoDB may be always writable (and thus fully available) with W = 1, but it has to return a set of possible states rather than a single canonical one. This is technically not eventual consistency, because obviously no single-threaded execution (non-concurrent history) would ever result in a set of states.

Wouldn't it be nice if we had full availability and (eventual) consistency? That's the promise of CRDTs:

replicas of any CRDT [eventually] converge to a common state that is equivalent to some correct sequential execution.

The key contract of CRDTs is: as long as you design your data structure in a specific way, it doesn't matter what network partitions you have, it doesn't matter in which order operations are applied, all nodes will eventually converge to the same state.

How does it do this? The idea is simple and genius. Recall that the various definitions of consistency are all about rearranging operations in some concurrent history (where calls can interleave) into a valid sequential history (where each call must immediately be followed by its response), subject to some other rules (follow process order, real-time order, etc.)

And the tricky bit is that in many data structures, the order of operations is important. The operations are not commutative.

If we have noncommutativity then getting a valid sequential history may be tough because you might violate an invariant.

This presents the two problems I mentioned earlier: firstly you must rearrange carefully so as not to violate the invariants of the object, and secondly every node must rearrange its operations in the exact same way

But of course, if your operations are commutative, then any arrangement of method calls is perfectly fine! No matter in what order you

set(1) set(3)

You can think of these things as existing in a consistency--availability Cartesian space, where the impossibility theorems constrain the possible spots in that space. Raft and 2PC are in one region: they guarantee strong consistency, but they're not available in the presence of partitions. Amazon's Dynamo with N=3 is in the middle; it (almost always) guarantees eventual consistency, and is (almost always) available. Dynamo with N=1 is further to the left: it's always writable, but forks more easily. And finally CRDTs are better than Dynamo with $N=1$ , fully available, _and _ always eventually consistent.

FAQ

What's the purpose of the `compare` function?

To map it to a monotonic join semilattice.

What's least upper bound?

What's a monotonic join semilattice?

Monotonic: you can only move "up" state, or don't move at all

a Join Semilattice is some ordered set where a LUB must exit. This ordered set is the set of all possible payload sate

What's a source pre-condition?

It's a "guard statement" that causes the query or update function to fail if a particular condition is not met.

What's a causal history?

The causal history of an object $x_i$ a set of all update operations applied to the object.

The operation can contain the snapshot or

Causal history op-based:

Downstream: executes after the source-local phase immediately at the source and asynchronously at all other replicas only if the downstream precondition is true

What's an "abstract state"? How does it differ from regular state?

Merge is idempotent and commutative by the properties of... messages may be lost, received out of order, or multiple times.

Why is it not possible to maintain a global invariant?

There is actually no non-negative counter. You can only enforce local invariants by definition

What's the importance of this paper:

The importance is that any data type that

What makes something a CRDT?

Query and add

"Only merge and compare must follow CRDT"

All merges must be idempotent , commutative, and associative to be a state-based CRDT.

What's a PN-Set? Do you need add and remove counter for each replica?

I think so.

Removing stuff is hard without synchronisation

When would you use CRDTs versus a more traditional synchronisation solution?

What's the process of designing or building a new CRDT?

I'll start with some primitives, LWW sets
Then compose higher level CRDTs on the primitives. E.g. 2P sets can be composed with grow-only sets.

Is it true that any CRDT can be represented as both state-based and op-based? Proof?

What are operational transforms?

What is quiescent consistency?

Quiescent consistency requires non-overlapping operations to appear to take effect in their real-time order, but overlapping operations might be reordered.

Pavel Shved's 2012 post on "Consistency Models Explained Briefly and this SO post are both very useful. Pavel's post needs a bit of background, but if you have it, it is a great post.

System design (and business case) for a EV charging station backend

2021-02-13T00:00:00Z

Introduction

Business case

There are 600,000 private cars in Singapore. Right now, 600 of them are (fully) electric.

By 2030, the SG government wants to have 60,000 chargers installed island-wide. Suppose there are 60,000 electric cars. These electric cars need to be charged at chargers. An average driver drives about 50km in a day. That means 10kWh. The business model has us charge 45c per kWh. So that means that each charger makes about $0.45 * 10 = \$4.5$ of revenue. Power from the grid costs $15c$ per kWh, so that's $\$3$ in profit per day.

A Toyota Corolla Altis gets an average of $15.4$ kms per liter source. At 50km, this is $3.25$ liters of fuel per day, which at a fuel cost of $\$2.25/liter$ equals $3.25 * \$2.25 = \$7.30/day$ . So the electric car driver saves almost 40% on fuel costs.

Suppose there are 60,000 electric cars (10% market penetration) and each of them gives us $\$4.50$ of daily revenue, then the total addressable market is $60,000 * \$4.50 * 365 = \$98,550,000.00$ , or almost 100 million dollars in revenue per annum.

Meta-question: when is it better to build your own v. outsource the software development?

Should ask Ben about this.

PlugIT (website in Finnish) offers a SaaS that makes the backend easy. It charges $21/month per charging station with an additional fee per charging transaction.

On the one hand, since PlugIT has already gone through all the hard work to develop this software, we would certainly incur a very high fixed cost to do the R&D in-house (especially if we're hiring top engineers, which we will want to).

On the other hand, in the long run building your own software would make financial sense. Suppose the company manages to install 4,000 chargers. Then at 21 dollars per charging station per month that's $\$82,000$ per month: enough to hire 5 or 6 full-time excellent SWEs to develop the front- and back- end and maintain it full-time. At the time of its acquisition, Instagram had 6 engineers supporting 30 million users, Even with a 50% volume discount this would be $\$40,000$ per month, enough to support three full-time engineers. I hear that PlugIT also takes a 1% extra transaction fee per transaction. Assuming just one transaction per day per charging station, that's $4.5c$ per transaction, which is $\$0.045 * 4000 * 30 = \$5400$ per month.

Also, an argument could be made that this is critical software infrastructure that should not be outsourced. Without the backend your company literally cannot make money (cannot take payment). It might not be a good idea to be reliant on a single company for all of your infra.

There are also more nebulous advantages wrt owning your own data, adding new features etc. For instance, you might be able to use the charging data to optimise charging patterns

I think it all depends on how much scale the company has. With only a few chargers it doesn't make financial sense to build your own software, but it does when you do have 4,000 chargers.

User story

The user first registers himself with the app and saves his credit card data. Subsequently he can drive up to any of the charging stations, press a button on the app to start charging, and plug the charger into the car.

On the backend once we receive a charge request this backend should send a "start charging" request to the charger via the OCPP 2.0 protocol. On the frontend the app will tell him how much energy he's used so far (ideally also battery percentage) and the cost incurred.

When the user finishes charging he again presses a button on the app which sends a "stop charging" request to the charger, which releases the safety lock and allows the user to unplug the charger from the car. A payment transaction is created on the backend and the user is charged appropriately.

Technical detail

OCPP 2.0 protocol

Questions to ask

Storing sensitive credit card data?

Server can never go down otherwise you literally cannot take payments and your business is dead in the water

WHat happens if the server stops responding? you must not have mechanical locks be software controlled

What comes next

What is a Golang interface?

2021-02-11T00:00:00Z

Thanks to Liang Jun for this.

Question

What does the line

struct Something {
    a []interface{}
}

actually mean? What is this interface thingy, anyway?

Explanation

Say you have a program that reads and writes to a database. Your program is passed a InMemoryStore object that contains the functions read() and write(). The database object now could be an in-memory store, but maybe in future you want to make it a persistent store? It would be troublesome if we had to edit the way our program calls the database

This is where interfaces come in - we define Database as an interface that contains read() and write():

interface Database{
   read()
   write()
}

Instead of having our program directly operate on the InMemoryStore object, we make InMemoryStore implement the interface Database:

struct InMemoryStore { // implements Database 
  read() {...}
  write() {...}
    
}

And have our program operate on the Database interface.

This means that in the future, if we want to swap the in-memory store to a persistent database, we just need to implement the persistent database to conform to the Database interface:

struct PersistentDB { // implements Database 
  read() {...}
  write() {...}
}

And the beauty of this is that our program doesn't need to be changed, we just need to pass a PersistentDB instead of an InMemoryStore to it.

Interfaces as multiple inheritance

Interfaces can also be used to implement something like multiple inheritance:

interface WithdrawalMachine{
  withdraw()
  
}
interface DepositMachine {
  deposit()
  
}
struct ATM { // implements WithdrawalMachine, DepositMachine 
  withdraw() {...}
  deposit() {...}
}

So what's []interface{} then??

In Golang, an object is considered to implicitly implement an interface if it implements all the methods of that interface. So let's say I have an interface and a struct:

interface MyInterface {
  A()
  B()
  C()
}

struct MyStruct {
  A()
  B()
  C()
  D()
  Z()
}

Because MyStruct implements all functions in MyInterface, Golang sees MyStruct as implementing MyInterface despite it never being explicitly declared anywhere.

Now consider the empty interface:

interface {

}

All structs in Golang implement it by default, since all structs implement "all" of its (zero) functions.

And therefore, a list of empty interfaces []interface{} basically means a list of Any objects.

What is the CAP theorem really about?

2021-02-11T00:00:00Z

Introduction

The NUS folks and I were discussing the Spinnaker paper while doing MIT's distributed systems course. We realised that we kept going around in circles talking about Spinnaker's guarantees: apparently DynamoDB is an "AP" system, but Spinnaker is a "CA" system? Is that right? What is a "CA" system? Can such a system exist? What does "partition tolerance" even mean?

The discussion got hopelessly muddled because we didn't really understand what the CAP theorem was really about. After very helpful discussions with Liang Jun, Julius and Wei Neng, I'm writing this post to (try to) shed some light on the topic, and dispel some common misconceptions I heard (and previously held!) during the discussion.

EDIT 2021-02-16: I just came across this wonderful CAP FAQ on the-paper-trial.org which has basically written everything I have here, but better! ^[1] I wish I had stumbled across this earlier. But I'm happy that what I have written here is correct and vindicated by this FAQ.

The CAP theorem is NOT "pick two out of three"!

One common mis-definition of CAP is:

You can only pick two out of consistency, availability, and partition tolerance.

This implies that there are "CA" systems, "CP" systems, and "AP systems". In fact, this couldn't be further from the truth. As a system designer you can't choose not to be partition-tolerant. Rather, almost all systems are (or can easily be) CA in the absence of partitions, and systems must choose to give up either C (Dynamo, Cassandra) or A (Spinnaker) in the presence of P. More on this later, but first, some definitions:

Consistency: What "consistency" is is actually super nontrivial

I often hear consistency defined as "a system is consistent if any request received by the system returns either the correct result or an error." But what is the "correct" result? It's too handwavy for my taste. Upon reading up more, I learned that there's strong consistency, linearisable consistency, timeline consistency, eventual consistency, and who knows what else.

I like the following two more formal treatments of consistency that show that there's actually a universe of consistencies. Jepsen has a clickable map and excellent definitions and Aphyr has a very readable treatment of the matter.

So which version of consistency do we want? The CAP theorem actually means linearisable consistency, or linearisability. This is quite a strong definition of consistency (and in fact it's often called "strong consistency"). The usual definition of linearisability is stated as follows:

Linearisable Consistency: A system is linearisable if all operations appear to take place atomically in an order consistent with the real-time ordering of those operations: e.g., if operation A completes before operation B begins, then B should logically take effect after A.

The real definition given by Herlihy and Wing (1990) is a tad more complicated. I'll go through it in the appendix.

Availability: thankfully easier to define

Availability: A system is available if every request received by a non-failing node in the system results in a response.

Note here "non-failing node!" This is key. We'll use this later.

Partition-tolerance: You need to define partition-tolerance with respect to something

A system must be partition-tolerant with respect to consistency or availability---it can't be just "partition-tolerant".

Partition Tolerance: A system is part ition-tolerant with respect to X if {condition for X} still holds when the set of nodes are split into any number of disjoint subsets that cannot communicate with any elements outside that subset.

Closely related, but not identical, is the notion of fault-tolerance:

Fault Tolerance: A system is fault-tolerant with respect to X up to F failures if {condition for X} still holds when any of F nodes fail.

Putting it all together: stating the CAP theorem properly

Almost all systems are CA in the absence of partitions or failures. We should therefore only talk about the CAP theorem in the presence of faults or network partitions. More precisely, when you have a network partition, you can only have one of linearisability (consistency) or availability.

Here's aphyr again:

Providing consistency and availability when the network is reliable is easy. Providing both when the network is not reliable is provably impossible. If your network is not perfectly reliable–and it isn’t–you cannot choose CA. This means that all practical distributed systems on commodity hardware can guarantee, at maximum, either AP or CP.

Paxos/Spinnaker/Raft are C in the presence of partitions, CA when the number of failing servers is less than half, and C when the number of failing servers is more than half.

More interestingly, because consistency is a spectrum (ish), as you start weakening the definition of consistency, you will eventually get availability AND "consistency" for some sufficiently weak definition of consistency.

Here's a nice map of the CAP theorem for different definitions of C. The ones shaded in red cannot be fully available:

The nicer version is available at the Jepsen website.

Partition-tolerance is NOT fault-tolerance!

Something I kept hearing during the discussions is that "network partitions are basically like node failures". That's not quite true! It is kind of true in a practical sense that if a node stops receiving and sending messages to the rest of the cluster because it's behind a network partition, that node might as well be failing.

But it does matter for the finicky definition of the CAP theorem, because availability is about queries on non-failing nodes. A node can be behind a network partition but not be failing, in which case that node has to return a value to be available (provided it can be reached by the client). Coda Hale made this mistake in his post You Can't Sacrifice Partition Tolerance., and made a correction:

... Therefore, considering a node failure as a partition results in an obviously inappropriate CAP theorem conclusion [emphasis mine].

He is correct: a dead (or wholly partitioned) node can receive no requests, and so the other nodes can easily compensate without compromising consistency or availability here. My error lies in forgetting that Gilbert and Lynch’s formulation of availability requires only non-failing nodes to respond, and that without ≥1 live nodes in a partition there isn’t the option for split-brain syndrome. I regret the error and thank both him and Dr. Brewer for pointing this out

Furthermore, there's a sense in which partition-tolerance is a stronger condition than fault-tolerance. If a system is partition-tolerant with respect to availability it is also fault-tolerant with respect to availability, but the reverse is not true. Systems like Raft or Paxos are fault-tolerant (with respect to availability), but not partition-tolerant, because that's literally impossible. Here's a rough sketch of a proof:

Claim: if a system is partition-tolerant with respect to availability, it is also fault-tolerant with respect to availability,

Proof:

Consider a partition where any F nodes are in individual disjoint subsets of size 1 and all other nodes are in another subset:

\{\{x_1\}, \{x_2\}, ... \{x_f\}, \{x_{f+1}, ... x_N\}\}

Since the system is partition-tolerant with respect to availability, every request received by any node in any subset results in a response. A fortiori, every request received by any node in the subset $\{x_{f+1} ... x_N\}$ results in a response.

Now consider if nodes $x_1$ through $x_f$ fail. The system is available if every request received by any non-failing node results in a response. Therefore, the system is available if every request received by any node $x_{f+1}$ through $x_{N}$ results in a response. But we know that every request received by any node in the subset $\{x_{f+1} ... x_N\}$ results in a response. Therefore, the system is still available despite the failure of $F$ nodes. QED.

Note that the converse is not true! Counterexample: Paxos/Raft are fault-tolerant with respect to availability up to $N/2 - 1$ node failures. But if one node is in its own partition with the client and you query that node, it will never be able to get quorum and will not return you a result--hence, it's not partition-tolerant.

Conclusion

TBD

References

Coda Hale (2010). You Can't Sacrifice Partition Tolerance.
Herlihy and Wing (1990). Linearizability: A Correctness Condition for Concurrent Objects
Jepsen (2018) Consistency Models
Kyle Kingsbury, a.k.a Aphyr (2014). Strong consistency models
Robinson (2013), on The Paper Trail. The CAP FAQ.

Appendix A: linearisability

Linearisability comes from the Herlihy and Wing (1990) paper. This is my attempt at explaining it simply because there's lots of notation, and my God do they do a bad job of explaining simply.

Linearisability is fundamentally about a history. We'll see how we will be able to expand this to objects and systems later. But what is a history? Let's use an example to demonstrate.

Suppose we have a queue object called r (why not call it q? Cause I'm gangsta that way). The queue object supports the operation enqueue x, which enqueues some value x, and dequeue, which returns the first object in the queue.

Most importantly, because we're talking about concurrency, there can be different processes calling functions on the same object. We use the notation r (Enq(x), B) to denote process B calling enqueue with the argument x on the object r.

Now these operations take time. We can split it up into two parts: the time when you call the function (enqueue or dequeue), and the time when you receive a response. Let's use the notation r (OK(x), A) to denote process A receiving a response to its latest function call with the result x.

So, a history is basically a log of all these operations, in time-ascending order (earliest first). Let's use an example to illustrate. Suppose processes A and B concurrently enqueue two values. While process A started first, process B's call ran quicker and returned earlier. Then the history would look like this:

r (Enq(x), A)   // process A starts calling first...
r (Enq(x), B)
r (OK, B)       // but process B gets a response first!
r (OK, A)

So that's a history. In particular, this is a concurrent history, because we can have function calls returning out-of-order: an earlier call might get a response after a later call. This is in contrast with a sequential history, where every call must immediately be followed by its matching response.

With those definitions down, we can then define linearisability.

Linearisability: An concurrent history is linearisable if you can rearrange it into a sequential history while following the following rules:

You must not violate the invariants of the object (e.g. a queue is FIFO, a stack is LIFO, etc.)
You must respect real-time ordering: If in the concurrent history you have Response_1 return before Call_2, you may never move Call_2-Response_2 before Call_1-Response_1.

Here's an example. The following history is linearisable,

r (Enq(x), B)
r (Enq(y), A)
r (OK, A)
r (OK, B)
r (Deq(), A)
r (Deq(), B)
r (OK(y), B)
r (OK(x), A)

because it can be rearranged into the following sequential ordering:

r (Enq(x), A)
r (OK, A)
r (Enq(y), B)
r (OK, B)
r (Deq(), A)
r (OK(x), A)
r (Deq(), B)
r (OK(y), B)

However, the following history is not linearisable,

r (Enq(x), A)
r (OK, A)
r (Enq(y), B)
r (Deq(), A)
r (OK, B)
r (OK(y), A)

because the only sequential reordering breaks one of the rules. If you moved forward r (OK, B) (the only possible rearrangement) you'd get the following:

r (Enq(x), A)
r (OK, A)
r (Enq(y), B)
r (OK, B)
r (Deq(), A)
r (OK(y), A)

But here $x$ is enqueued first, but $y$ is dequeued first, which violates the FIFO principle.

Note that if you didn't have rule number 2 (that rearrangements must follow the time ordering) you could get a valid history:

r (Enq(y), B)
r (OK, B)
r (Enq(x), A)
r (OK, A)
r (Deq(), A)
r (OK(y), A)

~~And in fact this is Lamport's sequential consistency: a concurrent history is sequentially consistent if you can rearrange it any-which-way to get a consistent sequential history.~~ (EDIT: this is actually serialisability, not sequential consistency. Serialisability allows you to arrange it any-which-way: sequential consistency allows you to rearrange it provided you follow a time order within a process. Linearisability needs you to follow a time order between all processes. Thanks to Julius and Zongran for pointing this out!) But this rearrangement is not allowed under linearisability because in the concurrent history r (OK, A) happened before r (Enq(y), B).

So what does this have to do with distributed systems and consistency?

The idea here is that a distributed system is made up of objects. Herlihy and Wing show that a system is linearisable (ergo "strongly consistent") if all its objects are linearisable.

But we've been talking about histories. how can objects be linearisable? Herlihy and Wing just whack definitions:

A set S of histories is prefix-closed if whenever H is in S, every prefix of H is also in S. A single-object history is one in which all events are associated with the same object. A sequential specification for an object is a prefix-closed set of single-object sequential histories for that object. ... A linearisable object is one whose concurrent histories are linearisable with respect to some sequential specification.

Eureka! Or maybe not, if you're not mega big brain. Let me try to unpack this in normal-people-speak without all the definitions.

Consider all the possible "valid" sequential histories of an object (that is, histories that don't break the invariants of the object, like popping a value before it's pushed, or dequeuing a value that wasn't first in). This set is technically finite since histories must be of bounded length, and you can't push unbounded values (maybe the max value is uint256 or whatever).

Basically, an object is linearisable if any valid concurrent history (a concurrent history that could actually happen) can be rearranged (following the rules) into a valid sequential history. So a queue is not linearisable, because we've just constructed examples of valid concurrent histories that are not linearisable. But a mutex would be linearisable, for instance.

So there we have it: the formal definition of linearisability. The worst thing is that it's not really helpful, because it's much easier to reason with the entirely equivalent "atomic" definition. But I hope it was interesting all the same.

I believe I explain linearisability better: I don't really get his explanation about "external communication being respected". ↩︎

Rao, Shekita and Tata (2011). Using Paxos to Build a Scalable, Consistent, and Highly Available Datastore [Spinnaker]

2021-02-08T00:00:00Z

Introduction

Overview of the paper

Partitioning and load balancing

Two-way replication doesn't work. Three-way replication works, but more complicated failures become possible. The Paxos familiy of protocols is widely considered to be the only proven solution when there are 3 or more replicas. Paxos solves the general problem of reaching consensus on the state of 2F+1 replicas while tolerating up to F failures.

Idea: three nodes replicate keys of a particular

A cohort of a key range $[i, j]$ is the set of nodes that contains its key range. So the cohort of the key range $[0,199]$ is the set ${A, B, C}$ .

Each node in Spinnaker contains several components. All components are thread safe.

There's a shared write-ahead log (shared amongst), each log record is uniquely identified with an LSN. Each cohort on a node uses its own logical LSNs.

The commit queue is a main-memory data structure that is used to track pending writes. Writes are committed only after receiving a sufficient number of acks from a cohort.

Committed writes are placed in a memtable, which is sorted and flushed to an immutable disk structure called an SSTable. (Sorted String table)

What is a write-ahead log?

Write-ahead log: before you perform an action, you save it to disk first

MemTable has the most recent state, we want to restore the state inside the memtable.

How does SSTable work?

If it's immutable, how is there a

What is Zookeeper?

Zookeeper is a "fault toleratnt, distributed coordination service".

The only messages exchanged between Zookeeper and a Spinnaker node are heartbeats.

Why don't we use Zookeeper for the entirety of Spinnaker?

What is the difference between strong consistency, eventual consistency, and timeline consistency?

Strong consistency vs eventual consistency

Strong consistency guarantees that all replicas available identical to applications.

Partition-tolerance:

Dynamo uses eventual consistency to provide high availability and partition tolerance. Dynamo is AP, so applications may see multiple versions of the same data.

Spinnaker is CA.

Spinnaker uses the following:

key based range partitioning,
3 way replication,
transactional get-put API with the option to choose either strong or time-line consistency on reads.

Not robust to network partition. It assumes that you will run it on a single datacenter where network partitions are rare.

"A data partition will be available for reads and writes as long as a majority of the nodes containing its replicas are alive."

Bigtable is CP apparently.

Optimistic concurrency control

OCC: the idea that data races are unlikely. You allow Assumption: low data contention.

To increment a counter $c$ ,

c = get(key, "c", consistent = true)
ret = conditionalPut(key, "c", c.value+1, c.version)

and this would fail if the c.version is not equal the later version.

Two-phase commit

A type of atomic protocol:

Commit-request phase (or voting phaes)
If all vote yes, then commit.

Two phase commit is overkill because if one node fails, we get an abort: not fault-tolerant.

2PC blocks when the coordinator fails.

Replication protocol

Two phases (in Paxos): leader election, then quorum.

When a client submits a write W.

when a leader gets an act from at least one follower, (so it has a majority 2/3), it commits W and writes to MemTable.

Periodically the leader sends an async commit meesage asking them to apply all pending writes up to a certain LSN to their memtable, effectively committing those writes.

Key guarantee: Anything written to the MemTable is guaranteed not to be lost.

Strong consistency vs timeline consistency

Strongly consistent reads are always routed to the cohort's leader, so they are guaranteed to see the latest value for W. However, timeline consistent reads are routed to any node in the cohort, so they may see a stale value for W.

Why is Spinnaker not partition-tolerant?

Where is the leader info stored?

How do we ensure that all committed writes are saved?

Leader election

Done on per-cohort basis, shared data in Zookeeper client

Two guarantees:

Will give a result
No committed writes are lost

How do we know when the leader is down?

Zookeeper keep-alive by heartbeat. "Ephemeral znode" of /r/leader will be deleted

How does the leader election in Zookeeper differ from the leader election in Raft?

A cohort will not lose committed data even if 2 out of its 3 nodes permanently fail.

What is partition tolerance really about?

Is a majority of nodes failing an availability thing or a network partition thing?

What is availability?

Every request received by a non-failing node in the system must result in a response.

Which part of

Is Zookeeper a CA system? Why don't you use Zookeeper as the data store?

What are the guarantees of Zookeeper?

Ongaro and Ousterhout (2014). In Search of an Understandable Consensus Algorithm [Raft]

2021-02-05T00:00:00Z

Introduction

This is the famous Raft (extended) paper, and here are my notes on it.

I read this paper as part of my self-study of 6.824 with NUS SoC folks. It may or may not be part of my goal to finish a Stanford CS degree in a year.

Overview of the paper

Raft is an algorithm for managing a replicated log. In the context of Raft a replicated log is a list of proposed changes to a state machine. Raft guarantees consistency (it's CP) and is fault-tolerant: the system remains available and consistent (linearisable?) ^[1] as long as a majority of nodes stay up.

It was designed with ease of understanding in mind: the authors claim that Paxos is difficult to understand and propose Raft as a more understandable alternative.

Safety properties of Raft:

Election safety: at most one leader can be elected in a given term. Leader append-only: a leader never overwrites or deletes entries in its log; it only appends new entries. Log matching: if two logs contain an entry with the same index and term, then the logs are identical in all entries up through the given index. (That is, if two logs agree on an entry in position i, they will also agree on all entries from 1 through i-1). Leader completeness: if a log entry is committed in a given term, that entry will be present in the logs of the leaders for all higher-numbered terms. State machine safety:if a server has applied a log entry at a given index to its state machine, no other server will ever apply a different log entry for the same index.

Interesting things about the paper

Questions I had about the paper

Leader election and the election timeout

A server remains in follower state as long as it receives valid RPCs from a leader or candidate. Leaders send periodic heartbeats (AppendixEntries RPCs with no log entries) to all followers in order to maintain their authority. If a follower receives no communication over a period of time called the election timeout, then it assumes there is no viable leader and begins an election to choose a new leader.

Why is it important that nodes disregard RequestVote RPCs before the minimum election timeout? What would happen if they didn't?

Find an example whereby nodes didn't disregard RequestVote RPCs before the minimum election timeout, and show that it is bad.

What is "copy-on-write"? Why is it important?

What is "linearisable semantics"? Why is it important?

Why does linearisable semantics imply no stale reads?

How do we prevent leaders from returning stale reads?

Leaders need to receive a heartbeat from a majority to make sure it's still the leader before returning any read data to the client.

Conclusion

I should think more about how to map these consistency guarantees to the actual safety properties of Raft. ↩︎

MIT's 6.824 Distributed Systems, Lab 1: MapReduce

2021-02-05T00:00:00Z

Introduction

Here I talk about what Lab 1 was about, what problems I faced when implementing it, and what I learned doing the lab.

This is part of my self-study of 6.824 with NUS SoC folks. It may or may not be part of my goal to finish a Stanford CS degree in a year.

The task

The task in Lab 1 was to implement MapReduce more-or-less faithfully as it was described in the original paper in Golang. The key difference is that all workers run on the same computer and all files are on the same local disk. One other difference is that while the original MapReduce paper describes the master pushing tasks to workers, we were told to have workers call the master using remote procedure calls (RPCs) to ask for tasks instead.

We were given scaffolding code and testing code which helped a great deal. All we had to do was write the master and worker code.

Reading the paper and grokking it took me around two hours, and implementing it took me about 6 hours. Overall, it wasn't too hard.

My implementation

Here's an overview of the process:

Periodically, workers call the Master with a RequestJob RPC.
Master checks current job status:
- if all jobs done: assign "exit" job
- elif there are idle Map jobs: assign Map job
- elif all Map jobs completed and there are idle Reduce jobs: assign Reduce job
- else: assign "null" job
When a worker receives a reply from RequestJob:
- if "exit": exit
- if "map": perform Map job on file at fileLocation. Save intermediate files as "mr-X-Y" where X is the index of the map job and Y is the (modded) hash of the key. When this job is done, call Master with ReceiveFileLocation RPC, giving the location of the intermediate files.
- if "reduce": perform Reduce job on all files at fileLocations, then save file as "mr-0-Y" where Y is the index of the reduce job.
- if "null": sleep(1), then RequestJob again.

When the master receives a ReceiveFileLocation RPC, it takes the file locations contained therein and saves them so that it can then pass them to the reduce workers.

The master does some bookkeeping to check when tasks are complete/have stalled, save file locations returned by ReceiveFileLocation, and so on.

Problems I faced, and how I solved them

Concurrency is the biggest challenge. The master is constantly receiving lots of different RPCs, each in a different thread. You can very easily have a race condition. For instance, the ReceiveFileLocation RPC causes the master to update its list of intermediate files. But these intermediate files might be read when responding to a RequestJob RPC. Since each RPC runs on a different thread, it's possible that the RequestJob RPC might read while the ReceiveFileLocation RPC is writing, which would result in inconsistent state.

Segfault because I didn't initialise arrays of objects

This was due to my lack of knowledge with Go. So I was declaring a := [][]string and then doing a[i] which throws a segfault, because the slice hasn't yet been initialised. Instead one should do a := make([][]string, N)

Deadlocks

When Liang Jun explained deadlocks to me I thought only an idiot would get into deadlocks. I guess that makes me an idiot 🙃.

The way he explained it is that deadlock happens when there's a function A holding the lock and is waiting for B to finish in order to unlock, while B is waiting for A to unlock so that it can start running. I thought "I'll never write code where this happens". Well--it turns out that you can easily do that with nested calls.

g() {
    m.mu.Lock()
    // do something ...
    m.mu.Unlock()
    return
}

f() {
    m.mu.Lock()
    g()
    m.mu.Unlock()
    return
}

Here f will acquire a lock, call g, then g will try to acquire the same lock. But it can't because f is still holding on to it. But f will never release the lock because it's waiting for g to complete! So we have a deadlock.

In this lab, g was GrabAllFileNamesForReduceJob and f was AssignReduceJobToWorker. Inside f there was a call reply.MapFileLocations = GrabAllFileNamesForReduceJob(..). It was very easy to make this mistake, because I thought "OK, so g() is reading state, we need to lock it". And if you do call it from a function that doesn't lock then you would indeed have to have a lock in g. But in this case since f locks, you cannot lock in g.

What I learned

This was a good warm-up exercise just to get to know Go better. Lab 2 is going to be very hard, so this practice is definitely welcome.

I didn't know what an RPC was until I took this course. I wish someone had told me this earlier, because my Python library I wrote for work is basically just reinventing an RPC library. I suppose this means that I can now write "designed and implemented custom RPC library with TCP sockets" in my resume...

I also went down the rabbit hole a little bit. In order to do this lab I had to learn about concurrency and mutexes. But in order to learn that I had to learn what the difference is between a thread and a process. And in order to learn that I had to know what a process even was -- which brought me back to operating systems and fork().

And, of course, I learned about MapReduce at a level that you just don't get from passively listening or reading alone. There's no substitute for implementation.

What I should have done better

Not a mistake per se, but I tried to stick very closely to the spec given in the actual MapReduce paper, but after doing a code review of Liang Jun's code I realised the code can be much cleaner if you don't follow it strictly. For instance, one thing Liang Jun did that I didn't was to make workers send an acknowledgement after a successful reduce task. I deliberately didn't do this because the paper says

"When a map task completes, the worker sends a message to the master... When a reduce task completes, the reduce worker atomically renames its temporary output file to the final output file",

implying that the reduce worker should not send a message back to the master. This entails extra complexity on the master because it has to have response handling code for the map workers, and file-checking code for the reduce workers. But as far as I can tell there's no good reason for reduce workers not to send back a completion too: having all workers send back a completion makes the implementation much cleaner.

Conclusion

Overall, I learned a lot from this lab. It wasn't too hard and I quite enjoyed writing in Go. I do kind of miss list comprehensions from Python, but I do like the simplicity of Golang. (There's a lot more to learn, of course!)

What I want to learn more about: Golang, Go's channels, interfaces, concurrent programming.

Dean and Ghemawat (2004). MapReduce: Simplified Data Processing on Large Clusters

2021-02-05T00:00:00Z

Introduction

This paper introduces MapReduce.

I read this paper as part of my self-study of 6.824 with NUS SoC folks. It may or may not be part of my goal to finish a Stanford CS degree in a year.

Overview of the paper

Interesting things about the paper

MapReduce gets map jobs

For some reason, the paper gets Map workers to send the list of the generated intermediate file locations back to the server, but doesn't do it for Reduce workers. I get it--there's not really a need for Reduce workers to send back since Reduce output files are saved on a shared file system while Map intermediate files are saved on the individual workers' storage. The drawback is that it results in additional work for the server to have to constantly check for the presence of output files. I compared my implementation to

I think this could be trying to save unnecessary network I/O.

How MapReduce avoids inconsistency

The problem is

With

Suppose you have a m

The paper prevents master from observing files that have been partially written by using temporary output files plus a atomic rename operation.

Self-studying MIT's 6.824 Distributed Systems

2021-02-01T00:00:00Z

Introduction

I'm auditing the NUS DYOM Distributed Systems course. Thanks to Wei Neng for introducing me:

I would do 6.284 if I were you

There's a class in NUS running now

Wei Neng told me to ask Joel, who then told me to consult Bernard, who managed to get me on board! Very pleased.

I was surprised at how progressive NUS was: they allow students to make their own module (they call it DYOM) and do it for credits. So this module is based on the MIT course 6.824: Distributed Systems. The difference between the NUS DYOM course and the MIT course is primarily that of scope. The original 6.824 is a graduate-level course, this is for undergraduates; the original course is taught by a professor, this is student-run; finally, MIT students usually do two modules a semester, while NUS students do five. So I think the plan is to do fewer labs (they will stop at lab 2C) read fewer papers, and have fewer lectures. However, I plan to finish all of the labs (up to Lab 3B and do the final project)
and read some additional papers on the side.

Structure of the course

From the MIT 6.824 course page:

6.824 is a core 12-unit graduate subject with lectures, readings, programming labs, an optional project, a mid-term exam, and a final exam. It will present abstractions and implementation techniques for engineering distributed systems. Major topics include fault tolerance, replication, and consistency. Much of the class consists of studying and discussing case studies of distributed systems.

We meet every Monday in person at SoC to go through a paper or proof. Each week a different person leads the discussion. We do the labs (1, 2A--2D, 3A, 3B) on our own time.

Papers covered in the course

The NUS students voted to cover a subset of the papers covered in the 2018 offering. Some of these papers are no longer being covered in the latest (2021) offering.

MapReduce (2004)
GFS (2003)
Fault-Tolerant Virtual Machines (2010)
Extended Raft paper (2014)
Zookeeper (2010)
CRAQ (2009)
Amazon Aurora (2017) (replaced with Pacifica (2008) in the 2021 offering)
Frangipani (1997)
Saltzer and Kaashoek (2009). Principles of Computer System Design: An Introduction (used in MIT's 6.033 course). Chapter 9, sections 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3
Spanner (2012)
FARM (2015)
Spark (2012)
Memcached at Facebook (2013)
COPS (2011)
Bitcoin (2008)
Blockstack (2017)
Analogic FS experience paper
Spinnaker (2011) (this was taken out in the 2021 offering, but we're reading it regardless)
Amazon DynamoDB (2007) (ditto)
Shapiro et al. (2011). CRDTs (not covered in the MIT course, but interesting and relevant)

I will be writing notes on each paper I read to serve as a reference for my future self, to keep me accountable, and to serve as a proof-of-work.

I also write notes in key distributed systems concepts: the CAP impossibility theorem and the FLP impossibility theorem.

Lab assignments in the course

Lab 1 implements MapReduce. I've successfully implemented it (took me about a day and a half to read and implement: 7--8 hours total?) My notes on Lab 1.
Lab 2 (A, B, C, D) implements the Raft consensus algorithm with log compaction. My notes on Lab 2.
Lab 3 (A, B) builds a fault-tolerant key-value store using the Raft consensus mechanism.
Lab 4 is either building a sharded key-value service, or creating a final project of your own. I think this is super cool, and I hope that someone in the class will be keen to do a final project with me.

Other things I expect to learn in the course

Programming in Go
The Go memory model
Concurrency, threads, locking, etc.

DIY-ing a Stanford CS degree in one year

2021-01-30T00:00:00Z

Introduction

Can we, through self-study, learn as much CS as a Stanford CS graduate would learn? And (stretch goal) can we do it in a year?

Spreadsheet tracking my progress here.

What would be the point?

If you wanted to optimise for getting a big job offer your time would probably be better spent grinding LC (for regular SWE jobs) or learning the ins and outs of C++ (for quant trading).

There is obviously no credential from doing this. That's not really important to me since I already have a degree from Oxford, so there would be very little (if any) marginal signaling value.

So what's the point? The main purpose for me is to fill up all the gaps in my knowledge. Having rubbed shoulders with folks like Bernard/Mark/WH/Julius/WN/Vig, I know there's still a lot I don't know, and I know there's a long way to go before I can match them. I want to fix this.

I also hope that some of these more "fundamental" concepts will be useful in my future work. In particular, I expect OS, networking, and databases to be particularly useful.

And of course, there's the joy of learning new things as well.

Why this? Why now?

Motivation is manyfold:

IMDA has asked me to come up with a personal development plan, and Eric recommended several courses
I want to fill in the gaps in my knowledge to become the best SWE I can be
The pandemic has forced many lecturers to upload their (very high quality) resources online and think about how to teach in a non-lecture style
CS is very easy to study online (many projects/lab-based with no final exams)
As a corollary to 4: If I self-study CS that frees me up to study maths/stats/ML for Master's

"Rules of the game"

The (very much self-imposed) rules for each module are:

Don't skip any of the lessons/work unless strictly necessary
Have a final project, paper, or something "real" to show for every course taken

A corollary to point 2 is that I should look for courses that are lab-based/project based. It's fortunate that I enjoy those courses better anyway.

There's no real need to do all Stanford courses: I'm happy to mix-and-match with MIT/other similarly (or more) rigorous offerings. For instance, instead of doing Stanford's CS 244B Distributed Systems offering, I am currently working through MIT's 6.824 Distributed Systems course with the NUS SoC folks.

Also, there's no need to strictly follow any "honour code" to do all the work myself. This is for three reasons. Firstly, since this is a self-directed project with no signaling value, the only person I would be cheating is myself. Secondly, doing work with someone else may give me the motivation to push through periods of low motivation/a part that I am blocked on. Finally, technical issues might stygmie my going it alone: most recently for CS140E/CS107E WSL2 didn't support UART so Wei Neng and I had to pair program.

Can you really do it in a year?

To graduate, a Stanford CS student must finish courses in all of these topic areas source: Stanford website:

Mathematics
Science
Technology in Society
Engineering Fundamentals
Writing in the Major
CS Core
Senior Project (Software Project)
Electives

This takes four years for a regular Stanford undergrad, but there are several reasons why I can do it quicker:

I've already done equivalents of several courses

CS109 Introduction to Probability for Computer Scientists: I've done a lot of this in my own probability course in uni.
CS106B Programming Abstractions: already know a lot of what is covered.
CS161 Design and Analysis of Algorithms: did Tim Roughgarden's course on Coursera.
CS229 Machine Learning: done Andrew Ng's course on Coursera.

We are at liberty to skip many of the non-CS courses

A regular Stanford undergrad needs to do compulsory Science modules, but I will skip them. I will also skip "Technology in Society", "Engineering Fundamentals", "Writing in the Major", because they are fluffy, and God knows I got enough writing in my major...

After skipping the science courses and fluffy modules, what's left for me to do is the following (strikethroughs denote courses I either already did, or already know from my own self-assessment):

CS 103 Mathematical Foundations of Computing
~~CS 109 Introduction to Probability for Computer Scientists~~
~~MATH 19 (Calculus I)~~
MATH 20 (Calculus II)
MATH 21 (Calculus III)
MATH Elective 1 (PHIL 151/CS 157 Computational Logic?)
MATH Elective 2 (linear algebra? real analysis)
~~CS 106B Programming Abstractions~~
CS 107E Computer Systems from the Ground Up
CS 111 Operating Systems Principles
~~CS 161 Design and Analysis of Algorithms~~
~~CS 194W Software Project~~

I also need to do seven electives. Some electives that particularly interest me are:

CS 140e Operating Systems
CS 144 Introduction to Computer Networking
CS 190 Software Design
CS 228 Probabilistic Graphical Models
~~CS 229 Machine Learning~~
CS 230 Deep Learning
CS 231n Convolutional Neural Networks
CS 244B Distributed Systems
CS 245 Principles of Data-Intensive Systems

The normal pace is for chumps

I took three years to do my degree, and by all metrics did it very well. That's $52 * 3 = 156$ weeks. But---thinking back---since the terms were only eight weeks long, and Trinity Term of final year doesn't really count, that's a total of $8 * 8 = 64$ study weeks. Of course some of the holidays were spent revising, but on the flip side there were also many days during term where I didn't work much if at all.

If one devoted yourself full-time to the pursuit I am confident that one could do the PPE degree in a year. At the very least one could give it a damn good shake.

I can do it: should I do it?

Similarly, if I devoted myself to this I'm pretty sure I could do it in a year. The question as always is whether I could do it but whether I should do it. Is it better to spend my time doing blub studies instead? Or more Leetcode? Or more side projects? Or write more essays that nobody reads? I'm not sure.

Making your habits zero-friction is a massive productivity hack

2021-01-28T00:00:00Z

TL;DR; I have found that aggressively minimising the friction of my tasks has allowed me to easily build and maintain desirable habits.

Introduction

We want to build and maintain habits because they allow us to convert our motivation (a finite and transient resource) into a more permanent routine. But I often find myself not sticking to the habits I've tried to build.

How can you stick to your habits? I have found that aggressively minimising task friction has been the most effective, and that even tiny decreases in friction (from almost-zero to zero) can make a huge difference in adherence.

In this post, I'll explain why habits are useful, define what it means to "aggressively minimise" the friction of a task, and give several examples of how aggressive minimisation helped me.

Aggressively minimising friction is a highly effective way to maintain one's habits

Define the static friction of a task as the time/effort/displeasure incurred to start doing the task, and the dynamic friction of a task as the effort required to actually perform it.

Scott Young writes that reducing static friction is by far more significant:

For a lot of tasks, the second cost reduction is far greater than the first. Flossing, for instance, hasn’t gotten any easier the hundredth time I’ve done it, but I have stopped thinking about whether I should do it.

He suggests starting a habit as a way to reduce the static friction:

If you read your fifty pages at lunch, every day, for three months, the next lunch break you’ll automatically start reading without having to decide whether to do it.

The idea is that if you succesfully maintain a habit you will decrease the task's static friction because that action will become habitual. This begs the question, however: how do I maintain the habits that I've started? I've found that aggressively minimising friction is by far the most effective method to do so.

Here are some examples:

I started working out much more often once I switched to a bodyweight fitness routine that didn't require me to travel to the gym, allowing me to maintain my daily workout habit.
I started vacuuming the floor of my room daily once we bought a cordless vacuum cleaner, because I no longer had to lug the heavy thing around and plug it into an outlet.
Simply leaving the guitar out of its case makes me practice 10x more often, because I could just pick it up whenever I got bored.

How do we aggressively minimise friction? I think there's not that much to it: Think about a habit that you're trying (and failing) to adhere to. Next, think about what you dread about/ what's difficult about starting the task. See if you can reduce or eliminate whatever that is.

There are no diminishing returns to decreasing friction

I have observed no diminishing returns to decreasing friction. Even a tiny bit of friction can spell the difference between a successful habit and an abandoned one.

Here are some examples of how small frictions make a difference:

Even on the very low-friction bodyweight routine, I am much more consistent in doing my "push" workout compared to my "pull" workout because the "pull" workout requires me to make the 2-minute walk to the fitness corner outside. I therefore bought gym equipment and put it in my room.
Putting Pleco (a dictionary app) on the first page of my home screen made me much more likely to look words up. The same was true for Anki, even though it literally takes one swipe to go from the first screen to the second.

Corollary: to break a bad habit, deliberately try to increase friction

Again, even a tiny bit of added friction can make a big difference!

Here are some examples:

I stopped playing as much computer games once I switched from Windows to Linux dual boot because it was a hassle to reboot the computer.
I wasted much less time on Facebook/LinkedIn/YouTube once I added a LeechBlock delay of just a few seconds to every link. The delay is only a few seconds long, but this is enough for me to close the tab immediately most of the time.

Conclusion

Reducing friction aggressively has outsized effects on habit adherence with little to no diminishing returns.

Group testing to save the world

2021-01-24T00:00:00Z

The year is 2021. The world has been transformed into a post-apocalyptic wasteland. COVID-19 has been cured, but one of the vaccines went horribly awry: some of the people who got the Oxford-Zeneca vaccine started frothing at the mouth, indiscriminately biting people, and chanting "Volker Halbach, Volker Halbach" before succumbing to their lack of qualia.

Two back-to-back pandemics were too much for the world to handle, and the remnants of humanity have been reduced to scavenging for GrabFood discounts.

But there is hope. A new screening test has been devised: a perfect test that can detect for sure whether someone has the virus in a matter of seconds...

You are an overworked civil servant at the Ministry of Health. Your boss comes in one day brandishing a couple of important-looking documents.

"New directive from the Minister---we need to test every traveler that comes in to the country. Can you write me a proposal by today?"

What choice do you have? You get to work.

0. The basic one

The proposal goes something like this: All visitors to the country will line up at a testing booth. When it's their turn they'll spit in a cup and then their secretion will be tested for the virus. If they test negative, they're free to go; if they test positive, they're whisked away to a quarantine facility ~~never to be seen again~~. You bang it all out and send it to your boss.

You get a call the next day. Your boss is aghast.

"We haven't got nearly enough tests to test everyone who comes!" he shouts.

"Can we order more tests?" you ask.

"No, the whole world is out of tests."

You take a deep breath to calm yourself.

"The Minister wants us to test everybody that comes in, right?"

"Yes."

"But we don't have enough tests to test everybody?"

"Yes."

A few choice words come to mind, but you (wisely) stay silent.

If your boss had any shame he'd have the courtesy to at least look sheepish, but unfortunately all senior civil servants have had that part of the brain surgically excised. In any case, he wants you to come up with a new proposal. "Can you find some way to test everyone, but with fewer tests? As few as possible?" You try to say something noncommittal but he's already hung up.

Clumps of your hair lie morosely on the table.

Testing everyone without enough tests for everyone? How is that possible?

You review the evidence. Preliminary findings suggest that out of every 1000 people, roughly 3 people will have the virus. Also, the test is perfectly accurate and infinitely-sensitive, which means that it always gives the right answer, and we can test as many people as we want with one test. (Since the test is infinitely-sensitive, you can test the tiniest drop of a person's spit, and it will give you the right answer).

Then you have a flash of insight:

Let's make one small change to the protocol. Visitors will have to queue-up-and-spit---just as before---but now TWO visitors will spit in one cup. This combined cup will then be sent for testing. Because the test is infinitely sensitive, it will test positive if either (or both) of the visitors have the virus. If the combined test comes back negative, then neither visitor has the virus, and they are free to go. If it comes back positive, then at least one of the pair has the virus. We then test each of them individually.

This two-visitors-one-cup protocol will save almost half of the tests because most of the tests will come out negative. Why is this so? The key here is that most travelers don't have the virus.

Let's use an example to illustrate.

Suppose we have 1,000 travelers and 3 of them have the virus. ^[1] If you test them individually then you will use up 1,000 tests. If you pair them up at random, however, one of the following must be the case:

There are three pairs with one positive person, and 497 all-negative pairs.
There is one pair with two positive people, one pair of one pair with one positive person, and 498 all-negative pairs.

In the first case, we will use 500 tests for each of the pairs. Three of those tests will come out positive and we will use an additional two tests for each of them. The total number of tests is $500 + 3*2 = 506$ .

In the second, we will again use 500 tests for each of the pairs, and use two additional tests for each of the two positive pairs, for a total of $500 + 2*2 = 504$ .

In either case, we've cut the number of tests needed by almost half! This method only works because most travelers are negative, and so most pairs will test negative. By contrast, if the infection rate was much higher and 800 out of every 1000 people had the virus then almost all of the pairs would test positive and you would do better just testing everyone individually.

If two is good, surely three is better, four even better, and ten double-plus-good better? We'll divide travelers into groups---of, say, ten--- and get them all to spit into the cup. Doing so could save us even more tests if most visitors don't have the virus. Assuming we only have three visitors with the virus out of our 1000 visitors, there can be at most three positive groups no matter the group size. Doing groups of 10 would mean at most thirty individual tests, for an upper bound of $100 + (3*10) = 130$ tests.

But of course, there's an upper bound to how many people you can put in a group. If you get all 1,000 people to spit into the same bucket (yucks) and test that you'll only have to do one combined test, but that test is definitely going to come out positive and you're going to have to test all 1,000 people individually. Then you haven't saved any tests at all.

We can calculate the optimal size for each group. The trade-off of having bigger groups is that you do fewer group tests that combine multiple people, but have to do more individual tests when a combined test comes back positive. Let's introduce some notation:

Number of total people = $N$
Number of infected = $K$
Number of groups = $G$
Number of people in each partition = $N/G = p$

In the worst case you have to do $G + (p * \min(K, G))$ tests.

Why is this so? In the first step you need to test the $G$ groups. And in the worst case, all $K$ of the groups will come out positive (ergo each positive group has exactly one positive person), and you'll have to test each of those $K$ groups individually which takes $p$ tests each.

We can prove by differentiation that the optimal number of people in each partition, $p^{*}$ , is given by $p^{*} = \sqrt{\frac{N}{k}}$ .

Using our example of $N=1000$ and $K=3$ , this would mean that the optimal group size is $\sqrt{333}$ which is something like 18.5. ^[2] If we use $p = 20$ instead (just so the numbers are easier to work with) you would need $50 + (3*20) = 110$ tests at most in the worst case, where you do 50 combined tests, then test the 3 positive groups of 20 individually.

That's quite a big reduction. By grouping people together into larger groups and testing those groups we can decrease the number of tests needed by almost 90%, from 1,000 to around 100.

You sketch out a diagram of how this larger group test approach works and email it to your boss. He makes a couple of quick scribbles on it and it's quickly distributed to Border Control:

For a while, all is well. You've done the impossible with your great leap of insight and your boss gives you an A in your performance appraisal. Your meteoric rise through the ranks of the Civil Service is all but assured, you think to yourself -- but alas, Fate is never so kind...

A few months later, you enter the office and find it in a state of upheaval.

It is pandemonium. Your colleagues are screaming, jumping on desks, and smashing file cabinets: files and papers litter the floor, and your Director is bawling his eyes out.

"What happened here?" you ask, and in between racking sobs he points to a stack of papers on his desk:

As part of the Digital Government Blueprint and the continuing economic difficulties caused by the post-pandemic pandemic, we call upon the Whole-of-Government to embark upon a Productivity Improvement and Digitisation Journey to increase WoG efficiency and reduce government expenditure...

Lesser men would have gotten hopelessly lost in the thick thicket of Bureaucrat-ease, but you realise the subtext immediately. In order for your boss to keep his year-end bonus all his charges must demonstrate a 25% efficiency increase ---which means you have to reduce the number of tests by 25%.

(At this point the reader might wonder if a fairly mundane requirement of a efficiency increase could lead to such an exaggerated scene. I congratulate such a reader for his good fortune not to have worked in the Civil Service).

You steal a furtive glance at your boss. He seems to have recovered very quickly and is now eyeing you expectantly. You try to tell him that you've already saved him 90% of tests but he's having none of it. He quickly shoos you out of his office, leaving you once again to do the seemingly impossible...

Your most recent proposal is to test groups of 20. If any group's test comes back positive, test every single person in that group. But must we test everyone individually once a group tests positive? Can't we group people together again into subgroups of, say, 5? And if that subgroup tests positive, can't we group people together again? And so on. After all, if we were able to save tests by splitting our original group of 1000 into sub-groups, maybe we can save even more tests by splitting our sub-groups into sub-subgroups etc.

Here's an example. We have a group of 20 that tested positive. We might then choose subgroups of 5 from that 20, and start testing the first subgroup of 5. If that subgroup tests positive we then split them again into sub-subgroups of 2 and 3, and if those sub-subgroups test positive we test them individually. As before, if any group/subgroup/subsubgroup... tests negative then we simply move on to the next one.

Does this actually save us tests? Let's think about how many tests we would use in the worst case. The worst case is when the positive tests are as "spread out" as possible: that is, in different groups/subgroups/sub-subgroups.

So we test 50 groups of 20, using 50 tests. In the worst case, three of the groups test positive.

When we get a positive group of 20 we split it up into subgroups of 5. There will be 12 such subgroups in total ( $(20 * 3)/ 5 = 12$ ). We test all 12, using 12 tests. In the worst case, three of those subgroups test positive.

Again, when we get a positive subgroup of 5 we split them up into sub-subgroups of 2 and 3. There will be three subgroups of 2 and three subgroups of 3.

Again we test all six, using 6 tests. Three of those sub-subgroups test positive, and because this is the worst case, all the positive sub-subgroups are of size 3.

Finally, we test each of those sub-subgroups individually, using $3*3 = 9$ tests, for a total of $50 + 12 + 6 + 9 = 77$ tests.

You've saved the day again! You send the improved proposal to your boss and pat yourself on the back for a job well done. You quickly update your Tinder profile: after all, nothing gets women more excited than "Achieved process improvements of more than 90% for the Ministry of Health".

But while things are still fresh in your head ---and knowing that your boss will probably try to squeeze more out of you--- you reckon that you should keep thinking of more improvements to continually dole out. You wouldn't want work to get in the way of your schmoozing, after all. So you keep cracking...

4. Can we do better by picking the right sub/subsubgroup size?

You know from part 2 that choosing the right group size makes a pretty big difference in the number of tests we need. It stands to reason that the subgroup and subsubgroup sizes also matter. But how should we pick these sizes? In part 2 we calculated the optimal group size (20) but in part 3 we picked the subgroup sizes arbitrarily (sizes 5 and 2). Can we do better?

It turns out that we should halve the size of the (sub)group each time.

Concretely, we start with a group size of 20 and first conduct a test on the first 10 of the 20 people (i.e. the subgroup size is 10), allowing us to eliminate either these 10 people (if the test came back negative) or the other 10 people (if the test came back positive). But what if both subgroups of 10 test positive? Given the small number of positive cases, the odds of getting two or more positive elements in a group of 20 are very low, so in the vast majority of cases we will eliminate half of the people. (But see a more rigorous treatment in the Appendix!)

After the first test 10 people remain. We conduct a test on the first 5 of the 10 people (subgroup size 5), allowing us to eliminate 5 people. We continue doing so until we are left with a single person --- this person must be infected. This is an idea very similar to binary search. (The proof of why binary search is optimal can be found in the Appendix).

5. Can we do better by picking the right group size (again)?

We previously showed in part 2 that the ideal number of people in each group is $p = \sqrt{\frac{N}{k}}$ . However, that calculation hinges on the assumption that there is only one level of grouping before we test each person individually. This isn't the case now, since we have many levels (specifically it's $\log_2 \frac{N}{p}$ levels, since we halve the number of people in a (sub)group after each test).

What is the new ideal group size $p$ ? Because we halve the size of the group each time, we need (at most) $\log_2 \frac{N}{p} + 1$ tests per infected individual. And since there are $k$ infected individuals, we need a total of $G + k (\log_2 \frac{N}{p} + 1)$ tests: $G$ initial tests, then $\log_2 \frac{N}{p} + 1$ individual tests. Using differentiation we find that the optimal number of people in each partition, $p^{*}$ , is given by

p^{*} = \frac{N}{k}.

Note how it is the square of the previous value

p^{*}_{old} = \sqrt{\frac{N}{k}}.

Importantly, this new group size is larger than the previous group size, which makes a lot of intuitive sense. Remember that the fundamental group size trade-off is that if you make your group size too large, you will expend a lot of tests testing every one individually (subgroups of 1). But in our refinements of the method we have managed to use fewer tests by choosing the right subgroup size. The optimal group size therefore increases. (And for our case of $N = 1000$ and $k = 3$ , the optimal group size is $p^{*}=333$ ).

6. Depth-first search: should we "reset" the search after we've found an infected person?

Previously, we were testing all 50 groups of 20 before testing the 12 groups of 5, and testing all the 12 groups of 5 before testing the 6 groups of 2 or 3.

There is another way to do it, though. We could test groups of 20 until we find a group of 20 that tests positive. Once we find a positive group of 20 we test groups of 5 from that 20 until we find a group of 5 that tests positive, and from that group test the groups of 2 and 3, and so on. Only when we find an infected individual do we go back up a level.

Here's an illustration of what that looks like:

This is called breadth- vs depth- first search. In breadth-first search, we explore all groups of the same size before exploring groups of smaller sizes (i.e. deeper levels). But in depth-first search, we make our way to the first infected person as quickly as possible, before backtracking to find the next infected person, and so on.

Does it matter whether we use depth- or breadth-first search? It turns out that depth-first search opens up some further optimisations. The idea is that once we find the first infected individual we can stop and "reset" the search, using fewer tests overall.

Suppose we conduct our tests, using the depth-first search procedure, pausing just after we have identified the first infected individual (in our example, it is the 13th person). But note here that we have no information whatsoever about the people behind the 13th person. More specifically, we know that persons 1 to 12 are not infected, and (of course) that person 13 is infected, but we don't know anything about whether any of the subsequent people are infected. This is because subsequent infected people don't affect any of the test results obtained so far by the procedure. This means that persons 14 to N are as likely to be infected as any other person we haven't tested: it is as if we never did any tests on them at all.

In our current procedure, after finding that the 13th person is infected, we backtrack and test the 14th and 15th persons together (size 2) then back out further and test the 16th to 20th person (size 5). But if you think about it, since persons 14 to 20 are as likely to be infected as any other person we haven't tested, we have no information about the 14th to 20th people, and as such have no reason to test them in small groups. You can think of it like the 13th person being positive "polluted" the group test of 20 people.

For this reason, if you did actually test the groups [14, 15] and [16, 20], both two tests will very likely be negative! And this is inefficient: it's similar to doing individual testing when we know that most people will test negative.

What should we do, then? The key insight here is to "reset" the search. Since persons 14 to N are equally likely to be infected, we arrive at a smaller instance of the same original problem --- there are $1000-13=987$ people that we want to test, of which $3-1 = 2$ of them are infected! It then follows that we should perform an identical procedure on these remaining people --- we recalculate the new value of $G$ (and hence $p^{*}$ ), and proceed with the procedure with those new values.

All the optimisations lead you to the final, optimal algorithm:

Set the optimal group size as $p = N/k$ . Pick the first $p$ people, group them together and test them.
If that group tests negative, remove all of these people, and repeat the algorithm with $N$ now set to $N-p$ .
If that group tests positive, take a subgroup of the first half of those people and test them:
- If that subgroup tests negative, clear them and test the first half of the other subgroup.
- If that subgroup tests positive, test the first half of the subgroup.
- In either case, repeat this process until we get to a single person.
Remove all the people before and including the first positive person, repeating the algorithm and setting $N$ and $k$ appropriately. Repeat this process until everyone has been cleared or quarantined.

You smugly file the optimal algorithm away, content in the knowledge that you'll be able to appease your boss and exceed your KPIs for many months to come. You're feeling very clever with yourself, but unbeknownst to you Hwang (1972) beat you to it almost fifty years ago with his paper A Method for Detecting All Defective Members in a Population by Group Testing (maybe you should have studied something useful instead of PPE). Over the next few months you spend your working hours writing florid poetry on the human condition, slowly moulding yourself into the Tang dynasty bureaucrat-poet all AOs surely aspire toward becoming.

THE END

Conclusion

In this post we went through Hwang's 1972 algorithm, which allows us to find all the people who test positive for a disease using only a small number of tests.

We kickstarted our intuition by realising that we can group people and test them together. If the number of positive people is small most of these groups will come back negative (and thus no further testing is necessary). This gave us a large decrease in the number of tests needed.

We further improved the method by noting that i) we can split groups into subgroups, sub-subgroups etc. when a group tests positive, and ii) the best way to split a group that tests positive is into two groups of half the size.

Finally, we showed that we simply arrive at a smaller instance of the same original problem once we find a positive person, and it is optimal to simply "reset" the search with smaller values of $N$ and $k$ rather than continue to split the groups/subgroups in the original search.

There are a few possible extensions to this piece. For instance, what if tests are noisy (they sometimes give false positives/negatives)? Then you can never have certainty, only a certain posterior probability. Also, this algorithm runs serially, not in parallel. You have to wait for the results of the group tests before testing the subgroups. What if (due to time constraints) you're not allowed to do that? Is there a better algorithm? What if you're allowed to do at most e.g. 3 rounds of tests? Can we find an algorithm that allows us to specify a tradeoff between the number of test kits needed verus the number of "rounds" of tests? Please let me know if you are interested in reading posts on these extensions, or have any suggestions for improvement.

Acknowledgements

I worked on this piece together with Bernard Teo Zhi Yi. The idea for this post came from Bernard, who is currently writing his thesis on group testing. He is very much the brains behind this piece.

A big thanks to Robert Heaton for reading a draft of this work and providing very valuable comments. Thanks also to CKY, Ed, Nick, Celine, Wilfred, and Joy for reading early drafts of this piece and giving me helpful comments.

Appendix

Why halving the group size each time is optimal

Each group that tests positive is quite likely to have only one infected individual, because there are more groups than infected individuals. Suppose you rolled twenty 100-sided dice (d100): the odds of getting one 100 is quite rate, but the odds of getting two or more 100s is vanishingly unlikely. In the same way, since an individual is very unlikely to have the virus, it is quite rare for a group to have an infected person, and extremely rare for a group to have two (or more) infected people.

Let's assume for now that each group that tests positive has exactly one infected individual. Our objective, then, is to determine which one of the 20 people in this group is infected with the minimum number of tests, and this is optimally done by halving the group size each time.

To see why halving the group size each time consider what would happen if we first conducted a test on 5 (instead of 10) of the 20 people in the group. If the test comes back positive, then we get to eliminate a whopping 15 people! But if the test comes back negative, then we can only eliminate 5 people. On average, then, what is the number of people that we will get to eliminate?

If you've answered "10 people", that's unfortunately wrong. Since the infected individual has an equal chance of being each of the 20 people in the group, it is more likely that the the infected individual is in the untested 15 remaining people rather than the 5 people we conducted a test on. This means that it's more likely that we will only get to eliminate 5 people, rather than 15. As such, the average number of people that we get to eliminate is less than 10.

We may make a similar argument if 15 people are chosen for the first test, instead of 5 or 10. The average number of people that we get to eliminate will also be less than 10.

Hence, testing exactly half of the group is the most efficient way to narrow down the group size.

Note that this method doesn't rely on knowing exactly how many people have the virus, only the expected value. If you knew that there were EXACTLY three people you could stop once you found three positives. For ease of exposition we illustrate an example where there are exactly 1,000 travelers and exactly 3 positive cases, but the method and the probabilities are very similar even if we work with expected values instead. ↩︎
You can achieve an effective group size of 18.5 by having groups of 18 and 19. ↩︎

From Third World to Utopia: Singapore could end poverty for our children forever, today

2021-01-18T00:00:00Z

Introduction

If the Singapore government earmarked $200,000 for each child at birth and let it grow until the child turned 18 before dispensing it, we could guarantee a basic income of ~$800 (inflation-adjusted!) every month in perpetuity without ever depleting the capital.

As the birth rate is 40,000 a year, guaranteeing such a basic income to all future children would cost the government 8 billion a year. Better yet, since the capital is never depleted, the capital from the first cohorts can be "recycled" for future cohorts once the first cohorts die out. The total cost is therefore bounded at $800 billion dollars (8 billion * 100 years lifespan).

I won't devote too much space here to defending a UBI; these arguments have been covered elsewhere. This essay aims to show that a UBI is actually well within our reach; I'll punt on the question on whether we'd want it to others. But if UBI really is as wonderful as its proponents say, and it is within our reach, we might just be one step from Eden.

The theory

Money makes money. If you put your money in the stock market or in bonds your money will always grow in the long run. So if you have a large enough sum of money you can withdraw a small enough sum of money such that the growth rate of your money exactly equals your withdrawal rate plus inflation. This means that that initial sum of money could theoretically last forever--ergo, a BI.

One glaring problem with this strategy is volatility: unfortunately, stonks don't always go up. However, we know that if you wait long enough stonks do always go up (in recorded history). And so the initial losses can be made back by future gains as long as your withdrawal rate is low enough. The good news is that pension funds and people who aim to retire early run these calculations all the time. Early Retirement Now find that over all 60 year time horizons a portfolio with 75--100% equities always manages capital preservation (if not growth). And in fact our government has it easier than this because sequence of returns risk is greatly reduced since the government is always both withdrawing (for older cohorts) and investing (for younger cohorts) at the same time.

So, in detail: if the government constitutionally guaranteed a 3% real interest rate (similar to what it does for CPF) for 18 years, $200,000 at year 0 would become $330,569 in year 18 (after adjusting for inflation). This would allow a withdrawal of $9,917 a year, or $826 a month. And this capital would never be touched. On death, the capital would be clawed back by the government, ready to be endowed to the next person.

Can we afford it?

How much is 8 billion? It sounds like a lot, but it's not that much, really. Singapore collected 2.6 billion in betting taxes alone (wild! I wonder how much comes from Singapore Pools and how much the casinos). Singapore collected 11.2 billion in GST, 12.4 billion in income taxes, and 16.7 billion in corporate taxes.

And our net investment returns contribution (NIRC) from our reserves last year was 18.63 billion. This is half of our long-term expected real return.

The Net Investment Returns Contribution (NIRC) to the government budget consists of two parts: up to 50% of the Net Investment Returns (NIR) on the net assets invested by GIC, MAS and Temasek Holdings, and up to 50% of the Net Investment Income (NII) derived from Past Reserves from the remaining assets.

The government put in 6 billion to the GST Voucher Fund last year. As the GST Voucher Fund is a permanent one, the government is likely to continue contributing that amount to the fund.

I really like the idea of the GST Voucher Fund (it's kinda-sorta UBI-lite). But if 6 billion for the GST Voucher Fund sounds good, how does 8 billion for the Free Money For Life For Everyone Born After 2021 Forever Fund sound?

Why not a more targeted welfare programme?

We can and should certainly do more for the bottom 10-20% of Singaporeans.

But I think there are political reasons why this might be more popular.

TODO: To flesh out here.

Something about universality being more popular Give the example of the UK's NHS being popular with the rich even though they are certainly paying more than they would under a privatised healthcare system Another point: we have to have a dream; a dream that excites people and gets them on board. Targeted welfare doesn't sound like a dream; ending poverty for our children does.

Why give money to the unborn and not the people who desperately need it now?

Three reasons:

The yet-unborn are going to have to bear a greater burden because of our shrinking population;
We can't quite afford to give UBI to everyone;
Parents spend a lot of time thinking, talking, and worrying about their children, so helping the children will help the parents.

TODO: To flesh out here.

Will a UBI not encourage indolence?

There's not enough research on UBI. But as far as the research shows, disincentives to work are minor.

But even if it did: what's wrong with indolence? Is our work-life balance today healthy? Should we have to put up with working 60-hour weeks and weekends? Don't we want our children be able to spend more time with their children? Isn't this something we should fight for?

A UBI isn't a panacea

I might have overpromised: it's not going to solve poverty. You'd still want comprehensive medical coverage (mandatory health insurance) for every citizen at the minimum.

But it'd go a long way: be it not having to be in arrears not worrying about not being able to make the rent of your rental flat, being able to afford the $500 co-pay (after MediSave deduction) of your cataract surgery...

What would it take to give every citizen a UBI?

35 billion annual payment forever, or 1.2 trillion set aside as a one-time payment. Our annual government revenue is 50b (with 18b NIRC), and we have maybe ~1 trillion as reserves, so we'd either need to double our reserves or devote all our reserves to UBI, which would not be financially or politically feasible.

The nice thing about this plan is that it's a "ramp up plan": it basically gives a small percentage of the population (those born in or after year X) a UBI every year. Eventually all people will be born in or after year X, and everyone will have a UBI.

Reference

Income for All Stanford BI Umbrella Review Measuring Poverty in Singapore

Radically new ways of living and working are for the first time available (again) to us

2021-01-10T00:00:00Z

https://levels.io/coworking-space-economics/
https://levels.io/the-greatest-migration/

Joe Rogan moved from Los Angeles to Austin, Texas this year. And as an influential person in his community, he was able to bring a big share of his friends of comedians and other celebrities over to Austin too.

Kanye West bought a $14 million dollar, 6,000-acre ranch in Wyoming and is building an eco-village with a farm, houses and schools where his kids will go. And there's more famous people who are planning the same.

The founder of Bali's most popular coworking space Dojo bought land north from Canggu near the beach and is building his own village. It will focus on creative, entrepreneurial, maker-type people and will feature tens of bungalow-style apartments, coworking spaces, a maker space and since it's in Bali: probably lots of swimming pools.

The idea with all of these is the same: get out of cities that feel isolating (think of Los Angeles' giant sprawl) and move to a place where you can be physically closer to your community of people: being able to walk to each other, instead of an hour car ride away.

It's decoupling, right? Key SWE principle. The best place to work is not always the best place to live (the tails come apart). And for the first time we can decouple these two things where previously you had to live where you worked.

We probably shouldn't start a cult of our own:

And if the communes of the 1960s taught us anything: it's that trying to re-invent society by building a new mini society in a village usually doesn't work out and sometimes even ends bad:

The 1960s teaches us we maybe the point isn't to try build entirely new artificially designed communities from the ground up, but instead iterate on the places that organically rise up for remote workers, improve them and solve the problems remote workers there have.

but I can see things like again trying to disconnect a little (but not completely!!) from the rest of society. There are two key takeaways:

In some sense I think we lose a lot when we scale up —> food/agriculture, schooling, etc. I learned so much from my tutors in Oxford because I had concentrated 1 to 1 time with them: qualitatively different from what a lecture or tutorial can give. In that vein i believe it's better to teach our children ourselves (or in a small collaborative team) rather than ship them over to school with 25 other children. Ditto with food: Big Ag optimises for shelf stability, size, aesthetics, but not for nutrition, taste, or healthfulness. The trade-off here is time/efficiency, but since we as a society are wealthier than ever before, it might be a good idea to make this tradeoff.
I think it's important to live in a community where you live near (walking distance to) all your friends. I think that was why I was so happy in Oxford. As I've mentioned before, I think the nuclear family + single-family home is a Bad Idea, not just because no man (or family) is an island, but also because it really prevents people from living walking distance from one another.

Planning to build a Discord spatial layer

2021-01-06T00:00:00Z

MVP

build a browser extension that edits the discord web client UI to include a canvas element

this canvas element should contain a multiplayer 2D environ

minimally, players should be able to move around the space with the arrow keys and this should update in real time (client-side prediction optional)

we should include SOME minimal discord integration element in the MVP to showcase its potential. i'm not sure what form this would take, building a discord bot is probably too time-consuming

what's the point?

how is it different from gathertown, second life, minecraft?

my thought: discord servers are already communities, with chill VC hangout rooms and the like we want to piggyback on this community, not try to supplant it. the only way we can get adoption is for people to use our software is to piggyback on an existing platform that people already like using. that's the main hurdle facing all the YC startups like gathertown rn i believe

also, discord allows us to write bots and there is already lots of bot games on discord that people enjoy playing: adding a spatial layer might be compelling to the demographic that likes playing botgames on discord.

what needs to be done?

check discord API: find a way for a our game to see which discord server its in and get a list of all the people in the server and a way to make sure that a user owns/controls his character
actual game state sync
canvas rendering of gameworld and players
handle player input (update state)
modify discord web client to include our canvas
package up as browser extension?

these can be done in parallel (figuring out how to share and update state, figuring out how to render game state, figuring out how to modify the web client) which makes it ideal for hackathon

Technical unknowns

How do we get a list of all the people in the server? is there a Discord API?
think about how a server admin would set up and reset the game state

further extensions

allow players to customise their avatar
grab text messages and display inside client
allow text commands to mutate gamestate

Worlds for hyperlocal communities

2021-01-06T00:00:00Z

Introduction

Main idea: a game world that acts as a thin "spatial layer" above any Discord server.

Players can walk around the game world which is basically just a 2D grid. It's a very "thin" extra layer over any Discord server: we'd hook into the Discord API to send and receive chat messages so all messages would go through the Discord server. Using this "spatial layer" would be completely optional: a person using the "spatial layer" can talk with someone who is just using the regular Discord client.

EDIT: we could do a browser extension piggybacking on the Discord web client. See screenshots:

What is the point?

Humans are naturally spatial creatures. There's something about having an avatar in a physical space that cannot be replicated with just text. WN and I were talking about why people like hanging out in MapleStory and just chat? I believe hanging out in Maple is compelling because it has something more than just chat: you can express yourself with your movement, fashion choices, your skills, your facial emotions. We want to bring this spatial richness to existing communities.

It's not easy to make friends on Discord or in Hack N Roll (you usually just stick to yourself or your hacker group) and the spatial component of this "Hack N Roll Game World" might make a world of difference: it helps discovery/sociality when you can walk up to an avatar and just chat with them in the public channel, or DM them. I believe it would increase the sense of community and encourage making friends. Not everyone will find this compelling, but I think I would.

Extensions

We might want to give players of this gameworld something to do: the equivalent of Maki Bot in Discord servers.

Why not just play an MMO? Or use Second Life?

(WIP section)

A discord server is already a community and you want to build something on top of it, not try to transplant/supplant the existing community
zero friction is important! how can you possibly convince everyone in your server to move to a game
very lightweight layer not a entire game is a feature
the layer is completely optional so you can still have the same community even when not in the game

Issues

Building a game world where people can move around in in real time might be too difficult/technically challenging

Other stuff we could do

This is an instantiation of the following meta-level principle:

meta-level: build X ( tinder, facebook, twitter, a game world, etc... ) for a hyperlocal community (the HnR discord server, Telegram group)
meta-meta-level: build Y that allows us to build some X for any hyperlocal community

I like the meta idea of building something over the hyperlocal community. Weineng suggested something like Tinder or FB but only for HnR participants.

Teleoperation: enabling remote work for nonremote workers

2021-01-04T00:00:00Z

The Poor ROI of Autonomy TL;DR: you don't really need full autonomous driving, even teleoperation (let drivers drive the trucks remotely from home) is a big win because drivers no longer have to sit around while their trucks are being unloaded and they can sleep at home instead of in their cabs.

This got me thinking whether we can do teleoperation with other stuff as well. Obviously we have those system whereby a very expensive surgeon teleoperates on a patient, but can we scale down the market?

YouTube video demonstrating teleoperation

Can we have a very cheap robotic arm system that can be mounted on a desk or some sort of factory floor and allow people to work at factories via remote control?

Any sort of job where:

you need something to be physically effected
range of motion is confined to a table
no human contact necessary

this would allow handicapped workers to contribute to the economy you could also employ NEETs and people from other countries

The cheater's chessboard

2021-01-03T00:00:00Z

Introduction

The idea is to build a custom chessboard that looks like a regular chessboard (with maybe clock at the side for time games) but can actually read the state of the board, calculate the best move, and secretly leak information about the best move to a player in the know.

How we'd do it

We'd have to wire up a Hall effect sensor for each square. A Hall effect sensor is like a reed switch which can detect the presence of a magnetic field.

The idea is that when the piece is lifted up, the circuit will cut off (0) amd when the piece is put down, the circuit will close again (1).

We won't be able to distinguish between pieces, but we will be able to guess it by assuming that the board is set up in the correct position at the start.

So for example if we detect that e2 square cuts off, then shortly after e4 square cuts on, it's most probably e2e4 move. We will have to handle things like castling, en passant, promotion (especially hard because a pawn can promote to any piece).

We would then feed this board state to a Raspi mounted within the chessboard, which would use a chess engine to calculate the best move and leak information to the player in a surreptitious way.

How do we leak information?

We need to leak information in a sneaky way Basically if you think about it we need to leak 12 bits: 6 bits starting square and 6 bits ending square uniquely determine any move (except pawn promotion, but we can worry about that later). For castling you can talk about the position of the King (moving two or three squares).

My idea is that we have clocks on either side of the board which act as time controls, but they will leak the information somehow, either by a different border thickness/ brightness level/something else.

Links

https://www.sparkfun.com/products/9312

https://sites.google.com/site/bergersprojects/reedcb/programming

Debugging my thesis

2020-12-25T00:00:00Z

Introduction

Since my undergraduate thesis was quite well-received, I thought it would be a good idea to publish it. To that end, Gabe and I have been working on my thesis. As a replication check both of us ran the code on our machines... both of our laptops produced the same data but they were different from the existing data generated by my old laptop.

In my thesis I generate 100,000 hypothetical districting plans, calculate four different compactness metrics (Polsby-Popper, Reock, Convex Hull, and my own Human Compactness) and see how they correlate with a measure of district homogeneity (Stephanopoulos's "spatial diversity" metric). On our replication runs we observed the following:

Spatial diversity metric unchanged
Polsby-Popper, Reock, and Convex Hull very slightly changed (around 1/10th of the values were different in the 11th or 12th decimal place)
Big changes in human compactness metric (from 0.7 to 0.9, that sort of thing)

This was pretty bad, and threatened to derail my research completely

What wasn't the bug?

What was the bug?

Because I didn't freeze the environment, we had to install the newest versions of the libraries.

There is one line of code in sample_rvps

all_rvps = all_rvps.to_crs("EPSG:4326")

which converts the initial shapefile to a new coordinate reference system (CRS).

For some reason, the to_crs function in the new setup gives very slightly different values, differing in the 11th or 12th decimal place. You can see it in the snippet below:

gdf = gdf.to_crs({"init": "epsg:4326"})
print(str(gdf["geometry"][0])) # on old machine: POINT (-73.59250979083568 41.01678126113871)
# This assert passes on my new machine but fails on my old one
assert str(gdf["geometry"][0]) == "POINT (-73.59250974534255 41.01678109722333)"
print("All asserts passed!")

This alone wouldn't cause that much of a discrepancy, but it just so happens that this tiny change causes a point which was originally out of the state line to be in the state line. This causes the points sampled by sample_rvps to change completely even with the same random seed, because all the points have been shifted down 1. Because the points have all changed, all the precalculated point-kNN values are all incorrect (because they were for different points). And this leads to completely incorrect results.

The other metrics all changed by a very very tiny amount.

initial_partition = GeographicPartition(
    graph,
    assignment="New_Seed",
    updaters={
        "cut_edges": cut_edges,
        "population": Tally("population", alias="population"),
        "spatial_diversity": spatial_diversity.calc_spatial_diversity,
        "human_compactness": partial(
            hc_utils.calculate_human_compactness,
            DURATION_DICT,
            tract_dict,
            duration_matrix,
        ),
        "polsby_compactness": polsby_popper,
        "reock_compactness": partial(reock.reock, state_shp),
    },)

We know the EPSG conversion is very slightly different in the new state. But there is no EPSG conversion in the Reock function!

My best guess (but I haven't verified this) is that there is an upstream change in how points are calculated. The Reock function calls the _generate_external_points subroutine to find all external points of the Census tracts so that it can draw the circumscribing circle of the district:

def _generate_external_points(row, geoid_to_id_mapping):
    '''
    For each Census Tract, generate the exterior points of that Tract
    '''
    exterior_points = []

    # print(row['geometry'].geom_type)
    if row['geometry'].geom_type == 'MultiPolygon':
        for polygon in row['geometry']:
            exterior_points += list(polygon.exterior.coords)
    else:
        assert(row['geometry'].geom_type == 'Polygon')
        polygon = row['geometry']
        exterior_points += list(polygon.exterior.coords)

    return exterior_points

I believe there might be a difference with polygon.exterior.coords. Perhaps there's something in the getting points from a shapefile thing that has changed. If the exterior points' coordinates are very slightly different, this will affect the Reock function.

What causes Polsby-Popper to change?

TL;DR: I have no idea.

I dug into the Gerrychain source code to look at the Polsby-Popper function:

import math

def compute_polsby_popper(area, perimeter):
    try:
        return 4 * math.pi * area / perimeter ** 2
    except ZeroDivisionError:
        return math.nan
                        
                        
def polsby_popper(partition):
    """
    Computes Polsby-Popper compactness scores for each district in the partition.
        """
    return {
        part: compute_polsby_popper(
            partition["area"][part], partition["perimeter"][part]
        )
        for part in partition.parts
    }

It gets area and perimeter from the Partition object, which takes in a graph, an assignment, and a list of updaters.

In 10_Process_Tracts.py we have the following code:

state_graph = Graph.from_geodataframe(state_tracts, ignore_errors = True)
state_graph.add_data(state_tracts)
state_tracts.to_file(f"./Data/Shapefiles/Tract2000_{fips}.shp")
state_graph.to_json(f"./Data/Dual_Graphs/Tract2000_{fips}.json")

Looking at the GerryChain docs for from_geodataframe:

Creates the adjacency Graph of geometries described by dataframe. The areas of the polygons are included as node attributes (with key area). The shared perimeter of neighboring polygons are included as edge attributes (with key shared_perim). Nodes corresponding to polygons on the boundary of the union of all the geometries (e.g., the state, if your dataframe describes VTDs) have a boundary_node attribute (set to True) and a boundary_perim attribute with the length of this “exterior” boundary. By default, areas and lengths are computed in a UTM projection suitable for the geometries. This prevents the bizarro area and perimeter values that show up when you accidentally do computations in Longitude-Latitude coordinates. If the user specifies reproject=False, then the areas and lengths will be computed in the GeoDataFrame’s current coordinate reference system. This option is for users who have a preferred CRS they would like to use.

So the areas and perimeters of the polygons have already been pre-calculated and saved. As far as I can tell, the areas and perimeters of the polygons are not recalculated. And there doesn't seem to be a reprojection either. Why then can the Polsby-Popper metric still give a different result?

Questions I still have

Why does to_crs give a very slightly different result? I created an issue + minimum reproducible example and posted it on the geopandas repo.

I am almost certain it is not the fault of geopandas but rather some upstream dependency. This is why reock and polsby-popper are affected as well.

What's next

Things I learned

Bit-by-bit replication should be done with conda list -e > requirements.txt. Not using conda list -e or pip freeze or the like is a mistake that I won't make again.

Conclusion

Report on CS107E Lab and Assignment 3

2020-12-21T00:00:00Z

Introduction

Lab 3

Writing a `strcpy` function

This was a pretty easy task

Debugging with GDB

We did a bit more debugging with GDB just to understand how the stack works. What exactly happens when a function is called? How does it allocate memory? And how does it jump back to the parent function?

More assembly

I put the following function into godbolt, an online C compiler. This function is quite instructive so I found it quite enlightening to understand it in detail.

void stack_array(void)
{
    char array[32];         // not initialized before read
    
    // printf("array[] = {");
    for (int i = 0; i < 5; i++) {
        printf(" 0x%x ", array[i]);
        array[i]++;     // increment starting from garbage
    }
    // printf("}\n");
}

This compiles into the following assembly code:

stack_array:
    stmfd sp!, {r4, r5, r6, lr}
    sub sp, sp, #32
    mov r5, #0
    b .L2
.L3:
    add r3, sp, #32
    add r6, r3, r5
    ldrb r4, [r6, #-32] @ zero_extendqisi2
    mov r1, r4
    ldr r0, .L5
    bl printf
    add r4, r4, #1
    strb r4, [r6, #-32]
    add r5, r5, #1
.L2:
    cmp r5, #4
    ble .L3
    add sp, sp, #32
    ldmfd sp!, {r4, r5, r6, lr}
    bx lr
.L5:
    .word .LC0
.LC0:
    .ascii " 0x%x \000"

The function starts at stack_array. After some preamble (more on this later) and initialising r5 to #0, it jumps to .L2. .L2 checks the value of r5. If r5 is less than 5 it ends the function and does some postamble before jumping back into the parent function using bx lr.

L3 is the main loop. What is happening in .L3:? r5 is i and r4 is the current value of array[i]. So what we do is we load the value in the address pointed to at r6 minus 32 into r4, we increment it, then we store it back into that address. Note that at every loop r6 is being added to Note also that the array is being added from the back.

`stmfd` and `ldmfd`

What do these mean? stm means store multiple and ldm means load multiple. If the stack is descending (ergo the stack starts from a large number and decreases) you use fd to do full descending, otherwise you use fa instead.

stmfd sp!, {r4, r5, r6, lr} stores the data inside those four registers into the first four words of the stack pointer, then decrements the stack pointer by 16 (4 * 4).

ldmfd sp!, {r4, r5, r6, lr} loads the first four words of the stack pointer back into those four registers.

Why do we do this? This basically saves the state of this register before we jump into the new function and restores the registers once the function has finished running.

So, the instruction

ldm r4!, {r0, r1, r2, r3}

can be represented by the following pseudocode:

r0 = *(int)(r4) 
r1 = *(int)(r4+4) 
r2 = *(int)(r4+8) 
r3 = *(int)(r4+12) 
r4 = r4 + 16 // writeback (16 bytes transferred)

In the variant without ! the writeback doesn't happen so R4 retains the original value.

Lab 3: Check-in

Explain how the lr register is used as part of making a function call. Which instruction writes to the lr register? Which instruction reads from it? What commands could you use in gdb to observe the changes to the lr register during execution of a function call?

The lr register is the link register. You store the

bl loop
add r1, #1
mov lr #500 @ this stores 500 into LR

loop:
@ do some stuff
@ ...
bx lr @ jump back to `add r1, #1`

One thing the prolog here stmfd sp! {r4, r5, r6, lr} does is to support nested functions. Since we only have one lr register, if we had a function calling another function the previous lr would get overwritten and we wouldn't be able to jump back to the grandparent function. That's why you store the lr at the start of the stack when a function is called and load it back into lr after the function has returned and that allows you to go back to your grandparent.

The bl instruction writes to the lr register but you can also use instructions like mov to write to the lr register since it's just another register (register 14). The bx lr instruction reads from the lr register. (it actually reads from any register).

We could use disassemble to look at the source code of the function currently in scope and also points to the current line. We can use info reg to look at the registers and step to go stepwise and keep using info reg to observe the changes in the lr register.

Why is it necessary to plug in both TX and RX for loopback mode to work?

If you don't plug in TX to RX you either don't receive the transmitted messages or you don't transmit any messages at all (if you plug in RX but not TX).

On a hosted system, executing an incorrect call to strlen (e.g. argument is an invalid address or unterminated string) can result in a runtime error (crash/halt). But when running bare metal on the Pi, every call to strlen (whether well-formed or not) will complete “successfully” and return a value. Explain the difference in behavior. What is the return value for an erroneous call?

Let's answer the second part first. Recall that the implementation of strlen iterates through incrementing memory addresses until we hit a \0. Now if we don't hit a zero strlen will keep increasing and the return value is undefined (it can be arbitarily high depending on the state of the Raspi).

Here's our best guess: when we run it on a hosted system the stack size is limited by the OS (see this SO answer)

But programs compiled with for instance C or C++ allocate a fixed size stack at program startup. The size of the stack can usually be specified at compile time (on my particular compiler it default to 1 MB).

and we're guessing that once you try to read values outside of that stack the OS will throw an error.

Assignment 3

string functions, stoi

Most of the string functions were easy but the stoi functions were tricky. WN and I implemented the functions separately as a learning experience. He finished much quicker than I did and was kind enough to review my code, which was very helpful.

xxx_to_base functions

start 2030 12/22 end 2215 12/22

We managed to write a passing string_to_base function. It was quite tricky. We sort of cheated by using printf in a REPL to debug our functions rather than using gdb.

sprintf

vsnprintf

printf

Report on CS107E Lab and Assignment 2

2020-12-20T00:00:00Z

Introduction

This was the second lab (of seven) of Stanford's CS107E course. It may or may not be part of my plan to study a Stanford CS degree in my spare time. CS107E is a computer architecture course where you build a simple personal computer shell on the Raspberry Pi bare metal (without an OS).

In this lab we disassembled a C program to understand how C programs are translated by the compiler into assembly. This also improved my understanding of assembly. I learned a bit about the stack We also learned a bit about Makefiles.

The assignment is to build a clock that displays the time on a seven-segment display. We didn't have the materials so we moved on to Lab 3 for now.

Using gdb

Add a breakpoint by using break. Use info reg to view the registers, disassemble to view the assembly code of the current function, and next and step to step through code execution.

Understanding Makefiles

The symbols that begin with $ and % in a pattern rule are handled by make using the following interpretations:

% is a wildcard symbol when used in a rule; %.o for example matches any file that ends with .o $@ refers to the left part of the rule, before the : $< refers to the first element in the right part of the rule, after the :

One more special variable $^ refers to all elements in the right part of the rule, after the :, which is to say all of the dependencies.

Breadboarding the clock circuit

TODO

Assignment 3

TODO

Report on CS107E Assignment 1

2020-12-19T00:00:00Z

Introduction

I managed to rope Wei Neng in to do CS107E with me. This may or may not be part of my plan to self-study a Stanford CS degree in my spare time. CS107E is a computer systems/computer architecture course.

CS107e: Computer Systems from the Ground Up is a variant of CS107 that teaches the fundamental concepts of computer systems through bare metal programming on the Raspberry Pi. Bare metal programming means you will not run an operating system on the Raspberry Pi and will make minimal use of libraries. This course also serves as an introduction to embedded systems. The course starts with the microprocessor and moves up to the C programming language, without skipping anything in between. The goal is to build a solid understanding of all aspects of how modern computers execute programs and how program development tools work.

Assignments build upon each other by adding more and more functionality to a core library. They culminate in a simple personal computer shell using a keyboard and display. Finally, students do a project of their choosing where they build a complete hardware-software system.

I took NAND 2 Tetris back in 2017 and I really enjoyed it; however it trades off depth for breadth in its whirlwind tour-de-force from bottom to top. I wanted to take a second course to solidify my understanding and fill in the gaps. And since I've always loved the idea of building software on the bare metal, this "build an OS from scratch on the Raspi" was very appealing.

And it comes highly recommended as well. This is what someone said on Reddit:

I just took 107E this quarter. I would highly, highly recommend 107E over 107. Be prepared for a giant time commitment and more work than 107... but take 107E. ...
That doesn’t even begin to touch the material. Frankly, in terms of time commitment and difficulty, 107E is more challenging than 107. For the first 4-5 weeks, you don’t even have print statements. All you have is, literally, a blinking LED light on your Pi to tell if you’re buggy or OK. You build literally everything from the ground up, with your own code, starting with no operating system and debugger, and there are times when things get hard. Crushingly hard, like spend multiple nights on a single bug hard.

Also, the grading expectations, at least nominally, are high. The course is curved (this quarter at least), but according to the website, to get an A you have to do 3 or more extensions; get the full system bonus (no major bugs in code); have an outstanding final project; trend towards + in code quality; etc. That’s a lot. Now, the requirements aren’t really that stringent, but this course will easily demand 20 hours a week and your best effort to get the extensions in and avoid falling behind.

But there’s no feeling quite like when you get your code working, or your console working at the end of the quarter. The workload is high — don’t let the “no exams” fool you, it’s more than 107 easily! — but the reward is incredible, and you get to program on a Raspberry Pi! You get to wire things up on breadboards, mess around with LEDs, transistors, resistors, break things, build things...

For our final projects this quarter, someone built a freaking Rubik’s cube solver robot! Another person built Pac-Man with NES controllers; another person, a piano game using touch sensors on a glove you wear like Michael Jackson; yet another person, a robotic tea kettle. 107? You’ll be banging your head against some inane heap allocator running on a machine you can’t even see, at the same time you’re building these cool things in 107E.

The topics covered are actually quite close to 107, as well. Naturally, the focus of 107E is a bit different, as hardware, not just the software that runs on it, is always part of the equation. You build software for your GPIO pins, then your memory allocator, then your strings library, then your graphics library and frame buffer handling... You’re building up a fully-working system in layers each week, step by step, working towards a console with commands you can type. But along the way, you’re also covering assembly, memory allocation, system architecture, processing, etc. and your understanding of those topics is no less than that of a 107 student.

Unfortunately because there seems to be some problem with my Raspi TTY, I had to share with Wei Neng and we work through the assignments together. This is very much against the Stanford Honor Code but I couldn't care less since i) I'm not a Stanford student, ii) pair work keeps me accountable, and iii) the only thing that happens if I cheat is cheating myself of my own learning, so I won't cheat.

What did we have to do?

Time taken: about 10 hours for everything all-in

In Assignment 1 we were tasked to write a "Larson scanner", which is basically just blinking some lights in an aesthetic-looking scanner pattern. The catch is that we had to write it in assembly. In order to make lights blink on the Raspi you have to write an "ON" bit to the GPIO memory address. Because it's assembly, there are no functions, while loops, or even variables. And in order to

What was hard about the task?

Something that was very hard is that you can't just use any number as an immediate. So for example if you try to do mov r1 #584 the code won't compile and it will give an error like invalid constant value or something like that. The rules are something that I haven't bothered to learn yet (it has something to do with barrel shifting---see the "What do I still need to know?" section)

The trick is to add byte-by-byte using an inclusive or command:

mov r1 #73
orr r1 #512

Another hard part about writing assembly is that it's kind of like a real time system. Every instruction is mapped to one line of binary code and that takes one clock cycle (I think). So everything is driven by the clock cycle and you can get precision on the order of milliseconds.

It was actually quite hard to do pulse-width modulation with a constant delay. Suppose you want to keep the LED on for 1 second. That's some number of clock cycles. However, when doing pulse-width modulation you want to flicker it off and on with a short delay between. The challenge is: how do you vary the brightness (ergo change the time the bulb is off vs on) while keeping the total time the LED on the same? Remember you don't have any sort of timer: you can only declare variables that tick down. You need a nested loop with the invariant that the nested loop must run for the same amount of time regardless of how bright or dim the LED is (i.e. how long the LED stays off). So that was pretty tricky.

What did I do well?

I enjoyed writing assembly code in N2T and this time was no different. The simplicity of assembly code is really quite nice. I could feel the assembly that I did in N2T come back to me. Even though the assembly language taught in N2T was a toy language, a lot of the concepts were transferrable. I found that I was able to write more simple/less convoluted assembly code than Wei Neng, almost certainly because of my previous practice.

What could I have done better?

We worked till 2am/230am until we were super tired, and we were already far from productive. I should have called a break earlier so that I wouldn't fuck my sleep schedule.

What did I learn?

I learned (a small part of) ARM64 assembly. I also learned about the concept of pulse-width modulation.

What do I still need to know?

I'm still quite foggy on why some immediate values are allowed and why others aren't. Read how ARM encodes immediates

Browse through the ARM instruction set manual

Final thoughts

I really enjoyed this assignment, and I hope I get to code more in assembly next time (or C at least).

Geolibertarianism with Harberger Taxes

2020-12-06T00:00:00Z

What is Geolibertarianism?

https://sites.google.com/site/justindkeith/home/geolibertarian-faq#geolib

Two Crazy Ideas: Public Land Exemptions and Self-Assessed Tax

https://www.reddit.com/r/georgism/comments/hf636b/two_crazy_ideas_public_land_exemptions_and/

https://medium.com/blockchannel/reimagining-property-fbce9d3832a4

Yay Stability Rents (introducing the idea of Harberger taxes)

https://www.overcomingbias.com/2017/10/for-stability-rents.html

https://ethresear.ch/t/highlight-robin-hansons-more-owner-forgiving-modified-harberger-tax/5720

One possibility is to set up a system where users pay the Harberger tax by prepaying funds into a contract, and there exists a separate mechanism where users can submit bids for any item of property; each bid must be fully collateralized. The tax rate that the current owner pays is computed based on the value of the highest bid. The owner has the ability to sell to the highest bidder at the bid price at any time (or the sale happens automatically if the prepaid tax contract runs out of money). This system is more favoring because sales are owner-initiated rather than buyer-initiated. One can also provide owners a further measure of stability by limiting the rate at which their tax can go up (eg. to one doubling per week).

This kind of owner-forgiving Harberger tax is not very useful for the “market-based eminent domain” use case: if Elon Musk suddenly puts up bids for an entire strip of houses going through a city to build a hyperloop there, then once some owners sell the other owners will know they can extract high prices and would be able to raise their prices (whereas with the traditional harberger tax everything would get snapped up at once). However, it seems very useful in cases where you’re not expecting buyers to want to buy a large set of items at the same time, eg. ENS domains.

Three levels of competency

2020-11-30T00:00:00Z

[Forwarded from lieu] that for many endeavours there are three levels of competence/knowledge

lieu, [22.11.20 21:03] [Forwarded from lieu] the first level is slavishly following / repeating dogma, like following a recipe when cooking, or following a recommended build in dota

lieu, [22.11.20 21:03] [Forwarded from lieu] the second level is being able to make smart substitutions, to riff on an existning recipe, e.g. make ingredient substitutions/know which can be elided, or when in dota adapt the build to your opponent, or when playing guitar know what chords can be substituted for others

lieu, [22.11.20 21:03] [Forwarded from lieu] the third level is knowing the fundamental underpinnings of your field and being able to build something ex nihilo, e.g. creating a new dish with any random arbitrary subset of ingredients, or creating interesting weird builds of your own, or going to the scientific literature to design a specific fitness routine to work for someone with one arm and one leg, etc.

A new method of training: can resting for hours between exercises maximise your gains?

2020-11-30T00:00:00Z

~ 1600 words

Introduction

After reading the scientific literature and thinking through first principles, I've been thinking about a training method that I believe (as far as the evidence goes) maximises hypertrophy but also happens to be very easy and painless. This training method deviates from most programs in two (three) main ways:

Advocates full body workouts, 5 days a week
Advocates long (~15--25 minute) rests between sets, and even longer rests (~hours) between exercises
(optional) Consider cluster sets (resting a couple of seconds in between reps) and blood-flow restriction training

While I designed the method for my very particular set of circumstances and preferences (I have a home gym and prefer squeezing in sets throughout the day rather than blocking out a large block of time), I believe this method of training could actually be optimal for muscle growth.

The key idea is to maximise effective volume and muscle protein synthesis while minimising metabolic stress and muscle damage. The long rests between sets/exercises brings to mind the Greasing the Groove (GtG) protocol but focuses on near-maximal sets with high (7--8) RPE to maximise muscle tension/volume.

Microfoundations: what makes muscle grow?

TL; DR: The science suggests that muscle tension (force production and stretch) is the primary (and possibly the only) driver of growth. There is little evidence to show that muscle damage ("microtears") or metabolic stress ("burn" from lactic acid, "pump") contribute to hypertrophy, suggesting that we should maximise muscle tension and minimise muscle damage/metabolic stress.

The scientific consensus used to believe otherwise. In Brad Schoenfield's seminal paper (2010)

he writes that

Mechanically induced tension produced both by force generation and stretch is considered essential to muscle growth.. [but is] unlikely to be solely responsible for hypertrophic gains associated with exercise.

He points to muscle damage and metabolic stress as contributing factors, and concludes that

Current research suggests that maximum gains in muscle hypertrophy are achieved by training regimens that produce significant metabolic stress while maintaining a moderate degree of muscle tension. A hypertrophy-oriented program should employ a repetition range of 6-12 reps per set with rest intervals of 60-90 seconds between sets.

But in this video Schoenfeld updates his model. He says that the evidence for an additive effect of muscle damage or metabolic stress on hypertrophy is really equivocal and none of it is compelling.

In 2017 Menno Henselmans gave a talk where he cites a lot of papers and claims that both muscle damage (microtears in the muscle fibers) and metabolic stress (acidosis, lactic acid buildup, "pump", etc.) are not drivers of muscle hypertrophy. I haven't read the papers he cited but I trust him. The TL;DW is that:

i) muscle damage is a necessary byproduct of muscle tension, but does not itself stimulate hypertrophy. Increased muscle damage causes greater muscle protein synthesis, but no net muscle growth; the increased muscle protein synthesis simply repairs the damaged muscles. Additionally, blood flow restriction training seems to be able to induce similar levels of muscle growth with little to no muscle damage.

ii) training to failure causes metabolic stress to rise exponentially but we observe little to no increase in muscle growth (S-shaped curve).

There seems to be no good evidence to believe that muscle damage or metabolic stress contribute to hypertrophy. (Also, muscle damage and metabolic stress both hurt, and I don't like to hurt). This leads to a pretty counterintuitive conclusion: in order to maximise hypertrophy, we should aim to maximise muscle tension/force production while minimising muscle damage and metabolic stress.

If we accept this conclusion, that implies that we should take long rests between sets and perform few sets of exercises for a muscle group at a high intensity. Both these adjustments minimise metabolic stress and maximise volume.

Why long rests?

TL;DR: It allows increased volume---the primary driver of growth--- and also works much better with my schedule/preferences.

Exercise volume can be defined as the product of total repetitions, sets, and load performed in a training session (Schoenfeld 2010). If we believe that muscle tension (ergo volume) is the primary driver of hypertrophy, the idea then is simple: if we take longer rests, we can do more volume, which means more gains! Here's Henselmans:

In conclusion, your rest interval matters primarily because it affects your training volume. As long as you perform a given amount of total training volume, it normally doesn’t matter how long you rest in between sets... It’s the total volume, not how you distribute it over time, that determines the signal for muscle growth. However, in practice, ‘work-equated’ doesn’t exist, as it’s just you, so resting shorter for a given amount of sets decreases how many reps you can do in later sets and thereby your training volume. This means for most people, resting only a minute or less in between sets is probably detrimental for muscle growth rather than beneficial.

The scientific literature doesn't seem to examine very long rest periods, probably because nobody has the time to sit around for ages doing nothing. But I do because I now have a home gym and can bang out a set in between Pomodoro timers of 25 minutes. It's unlikely that resting for a long period would lower hypertrophy precisely because metabolic stress and/or muscle damage don't seem to be important for growth (but see the Appendix for concerns about "cooling down").

In essence, hitting one muscle group a lot in one session with not-very-long rests between sets both decreases volume causes a great deal of metabolic stress compared to doing the same exercises over several days AND taking long rests between them. The best part is that if you do them spread out you can do more volume which means more gains---AND you hurt much less.

Why a 5-day full-body workout?

TL;DR: Training each bodypart more frequently maximises muscle protein synthesis.

Damas et al. (2015) found that muscle protein synthesis (MPS) in trained individuals returns to baseline just 12 hours after working out, which means that in order to keep muscle protein synthesis at its peak for a particular muscle one should be working that muscle every single day---possibly even two times a day for elite athletes.

Additionally, muscle growth is a hyperlocal phenomenon: elevated MPS in your chest isn't going to make your arms or quads bigger.

This is why we should work full-body. Since

MPS is hyperlocal,
MPS drops off 12 hours after working out,
MPS causes muscle hypertrophy,

it therefore follows that we should work each body part every 12 hours (or as often as possible, anyway) to maximise MPS and muscle hypertrophy.

Additionally, there's the concept of "maximal effective volume": there may or may not be a hard cap of how much volume a muscle group can be worked in a session to see more gains. See Menno again.

Both these factors combine to suggest that body-part splits are suboptimal: if you really demolish one body part for a workout then not only do you get fatigued and can't really maximise volume/get the full mind-muscle connection, but that massive volume load can only really cause a spike in muscle protein synthesis for 12 hours anyway.

Empirical evidence seems to support this: Zaroni et al. (2019) found that a 5-day per week whole-body workout gave significantly greater hypertrophy on trained men.

The purpose of this study was to compare the effect a split training routine with muscle groups trained once per week (SPLIT) vs. whole-body split training routine with muscle groups trained 5 days per week (TOTAL) on neuromuscular adaptations in well-trained men... TOTAL induced a significantly greater increase in MT of the forearm flexors and vastus lateralis (p < 0.05)... TOTAL may confer a potentially superior hypertrophic effect.

Why five days, and not six or seven? I'm not sure (and would have to do more research on this) but it could be a way to reduce accumulated fatigue

Why cluster sets/blood flow restriction training?

This part is more speculative.

Cluster sets (sets with built-in, short rest periods of 5-20 or so seconds) decrease metabolic stress and allow you to get more volume in. If you rest between sets you can lift more weight more times --> more volume --> more hypertrophy.

But take note that in order to maximise muscle fiber recruitment one might want to work at a high intensity (high % of 1RM): you don't want to do cluster sets with too low weight as that might only recruit the lower-threshold muscle fibers. See the section in the Appendix (under "What about the notion of 'effective reps'") for details.

Blood flow restriction training might decrease muscle damage which might be useful, but this is really speculative.

Questions I still have

Will long rests make your muscles cool down? Will you have to warm up between sets?

Anecdotally, powerlifters seem to rest for ~15 minutes between sets without having to warm up between sets so maybe this is OK? Not sure.

You might need to elevate your core temp again between exercises if you've rested a long time between them.

Sample program

TODO

5 days a week full body hitting 10-20 sets of volume for each body part Dumbbell + calisthenics only because that's what I have.

Might take a look at Jeff Nippard's video for some ideas.

Chest: pushup variations, incline DB press, flat DB press,
Arms: skullcrushers, bicep curls, tricep extensions
Back:
- Lats: unilateral DB rows, humble rows, archer pullps, weighted pullups,
- Traps should be worked by heavy shrugs
Shoulders: side and front delt don't need to be isolated and will be worked with seated or standing DB press
- Rear delts should be isolated with ~15 sets a week: unilateral bent over external shoulder rotation, band face pulls,
Abs: leg raises, ab slides, double leg thrust
Legs: lol what are these

Mix between unilateral and bilateral execrises, calisthenics and dumbbells

Monday

Chest: DB bench
Shoulders: Seated DB OHP
Back: DB rows
Arms: Hammer curl
Abs:

Tuesday

Chest:
Back: weighted pullups
Shoulders:
Arms:
Abs:

Wednesday

Chest: floor press
Back: weighted shrugs
Shoulders: Lat raises
Arms: concentration curls
Abs: ab roll

Thursday

Chest:
Back: archer pullups, face pulls
Shoulders: Unilateral bent over external shoulder rotation
Arms:
Abs

Friday

Chest: DB bench
Back: Humble row
Shoulders:
Arms:
Abs

Conclusion

TODO

Appendix

Things that don't fit in the main article

Notes I took from Menno's video

Three primary drivers have been proposed for hypertrophy:

Mechanical tension
1. Mechanotransduction (tension on muscle stretch)
2. Myogenic signaling (IGF1 IL6) signals
3. mTOR signal integration and translation initiation --> (MGF Mechanogrowth factor)
4. MPS muscle protein synthesis
5. Myonuclear addition
Muscle damage: no compelling rationale.
- microtears repair, satellite cell activation, inflammatory response, repaired myofibre and gets larger.
- muscle damage triggers muscle growth, but not NET muscle growth. Cast doubt ... muscle damage is something we want to avoid. (Which study?)
- Another study: triggering the "repeated bout effect" in the treatment group. Muscle growth same with big difference in muscle damage.
- Long believe ecc > con for muscle damage & growth. KAATSU: no muscle damage, adding occlusion gives more muscle growth but no more muscle damage. Creatine (seems to) decreases muscle damage yet improve muscle growth.
Metabolic stress Accumulation of metabolic byproducts in the muscle: lactate, phosphate, hydrogen ion, along with hypoxia. Why might it work? Increased inflammation, fiber recruitment, cellular swelling (ergo PUMP: extracellular water enters the sarcoplasm outside the myofibers) HIIT high metabolic stress and muscle damage but little growth? (Cypryan 2017)

Rest intervals: broscience says long rests for strength, short rest If you have people resting longer, more muscle growth, you do more volume, put more tension

Henselman and Schoenfeld: volume > metabolic stress

Low reps high reps doesn't matter, mechanical tension

Training to failure: metabolic stress rises exponentially but no/little increases in myuscle growth (S-shaped curve )

Cluster sets: rest a couple breaths between sets (lot more work done): same growth, much less metabolic stress. --> Oliver et al. 2013 (check this study methodology though)

KAATSU works not via metabolic stress, but rather making exercise harder, causing more muscle activation + more mechanical tension

Conclusion: Mechanical tension is the only (or by far the primary) cause of hypertrophy!

What about the notion of "effective reps"?

"Effective" reps seem to be a thing (ergo you maximise hypertrophy by going close to failure, with diminishing returns the closer to failure you go). Some have claimed that this is due to a metabolic stress effect. However this is probably a muscle recruitment effect (recruiting higher-threshold fibers) and not a metabolic stress one.

Goto et al (2005) found that people who took a 30s rest in between 10 reps of lat pulldowns got less growth than people who did them all in one go. This is readily explained by the muscle recruitment hypothesis though. As you go closer and closer to failure you recruit more and more fibers, and recruiting more fibers means more hypertrophy. So I believe that resting 30s meant that participants continued to activate their low threshold fibers. They were not able to get full motor recruitment and thus had less hypertrophy.

BUT see also this correction by Grgic and Schoenfeld (2019): (https://physoc.onlinelibrary.wiley.com/doi/pdfdirect/10.1113/JP278627) looks like the evidence is equivocal on whether or not load (% of 1RM) is an important determinant of muscle fiber activation vs hypertrophy.

Also, a more recent study by Carroll et al. (2019) found that not going to failure resulted in greater hypertrophy.

In the interview I posted above Schoenfeld mentioned SOME evidence that acidosis prevents contractability in the fast twitch fibers. Less contractability means less fiber recruitment means less hypertrophy. This suggests that decreasing metabolic fatigue (ergo less acidosis) can directly increase hypertrophy.

He also says that there is no study yet on whether moderate or low sustained muscle damage or metabolic stress has an additive effect on hypertrophy, and part of the reason is that it's really difficult to isolate the individual effects in an RCT.

Convergence is (finally) coming

2020-11-30T00:00:00Z

lieu, [26.11.20 14:28] i think the future will be

you carry a work phone and a personal phone and there will be laptop/desktop/tablet "shells" completely dividing up form from function You only upgrade one device

You can keep the shells and peripherals the same

That is a key value prop imo

And the best thing is that those shells can be really any form factor you want really They don't even have to be restricted to laptop tablet desktop Could be anything That's what i mean by decoupling Conceivably in the future you might have a touchscreen wall let's say Just plug your phone into the dock And then the entire wall mounted screen becomes a massive tablet If you want to do big screen gaming plug it into your TV for consumer .. the way forward is to wait for apple to do it

Microsoft doesn't have the solution because you need to control both mobile and desktop and apple is the only person who does Google has chromebooks which are shite Microsoft king of desktop only so apple is the only one who could pull it off you have a phone which is your primary computer and you can plug it into anything

we are decoupling the form of the device from the brain of the device just like i plug my laptop into a 28 inch monitor this is just taking it ot the next step you have a phone that can plug into a tablet/laptop/desktop shell or "dock"

Your Phone Could Be Your Laptop. Here's Why It Won't.

How to set up bidirectional communication between a Docker container and its host machine using TCP sockets

2020-11-15T00:00:00Z

Introduction

(I wrote this to document the specifics of a feature I shipped at IMDA. I have reproduced it here because it may be useful to some (especially future me), but I haven't intended it for public consumption. So it's probably not going to be well-motivated or interesting to you, and you may be missing some critical context: but read on if you want to know how to set this up for yourself ).

I wrote simple sockets code to allow a host machine and a Docker container running on it to communicate bidirectionally. I needed to do this because our machine learning model ran on a Docker container and we needed to send data to/receive data from it.

The problem

We want the machine learning model to be in a Dockerfile to make setup and configuration easy. But this introduces a new challenge: how do we send data from the Jetson (host) to the machine learning model (Docker container)?

If the code for the machine learning model were not in a Docker, we could simply import the module into the webcam stream code, something like this:

# webcam_stream.py
import ml_model
img = get_img_from_webcam_stream()
bounding_boxes = ml_model.detect_objects_in(img)

But we can't do this now because ml_model is running in a Docker container. We need some way to pass the img from the webcam stream into the Docker container, and get the bounding_boxes back. So how? I thought a lot about this. There were a few different candidates:

Spin up HTTP server on the Docker container and make requests
Use a message broker like Kafka or RabbitMQ
Mount a host directory on the docker and read/write to files on that directory
Expose a port and build bidirectional communication over TCP

It seemed like a very simple task to send data to and from the same machine, so I wanted to avoid the overhead of spinning up a HTTP server or message broker. This left the last two options.

Writing to and from files in a directory

Suppose you have a webcam_stream.py and ml_model.py that both have access to a shared folder. If both Python files watch the folder for changes we can get kind-of real-time communication. Let webcam_stream.py write frame_1.ndarray to the folder. Then ml_model.py can read that file in, detect the objects in that first frame, and then write a frame_1_bbox.ndarray file to the folder. webcam_stream.py can then read that file in.

This is a pretty simple idea: the downsides are that i) file I/O is slow, and ii) you will have to write code to handle deleting old files. It also didn't seem very elegant to have to constantly create and delete files. Therefore, I chose not to use this approach in the end. But since everything in Unix is a file ^[1], couldn't we use Unix sockets?

Building bidirectional communication over TCP

Suppose we had an open port on localhost:6000 and a way to send and receive data over that connection. Then we could make ml_model.py constantly listen on that port for image data from webcam_stream.py. After receiving the image data it would then run it through the ML model and send the bounding boxes back through the same connection.

Some pseudocode here:

# webcam_stream.py
while True:
    img = get_img_from_webcam_stream()
    bounding_boxes = send_img_to_ml_model(img, port=6000)

# ml_model.py
while True:
    img = recv_img_from_stream(port=6000)
    bounding_boxes = detect_objects_in(img)
    send_img_to_stream(bounding_boxes, port=6000)

How to build bidirectional communication over TCP

I used a low-level Python library called socket. This library directly calls the Unix socket syscalls (again, something I don't quite understand).

We first establish a connection using socket.connect before sending any data over.

Server-side: First we start a listener on the server side. We bind the socket to an open port (in this case 6000) and start listening for connections forever in a while loop, calling socket.accept to accept incoming connections on that socket. Something to note here is that the socket.accept call actually returns a new socket when a client connects to it: a new socket is created for the connection ready to send and recv, and the original socket continues to listen for new connections ^[2].

Client-side: The client side is simple: we just use socket.connect to connect to the remote socket.

Here's what that looks like on the server side:

# ml_model.py
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_address = ('localhost', 6000)
    sock.bind(server_address)
    sock.listen(1)

    try:
        while True:
            print('waiting for a connection')
            connection, client_address = sock.accept()
            ...

And on the client-side:

# webcam_stream.py
server_address = ('localhost', 6000)

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    sock.connect(server_address)
    ...

Once a connection has been accepted we can start the bidirectional sending. Sadly it's not as simple as sending the image and receiving the bounding boxes. There are two main complications: serialisation and deserialisation of the data, and knowing when the message has ended.

Serialising and deserialising the data

The first main complication is how to send numpy arrays over the TCP connection, because a TCP connection only accepts bytes. One way to do it would be to simply convert the numpy arrays into a string, convert that into a bytestring using bytes(), and send that.

There's a better way to do it though. I use Numpy's built-in np.save and np.load to write and read to a BytesIO object, which is basically a file AFAIK. The BytesIO file can simply be file.read() and sent over.

Take note that writing to a file places a pointer to the end of the file (not sure how this works) so we'll need to do file.seek(0)to seek to the start of the file in order to read the file with file.read.

Here's what that looks like:

message = BytesIO()
np.save(message, img)
message.seek(0)
sock.sendall(message.read()) # sock is the connected socket

When receiving a response we simply use file.write() to write to a new BytesIO object. and np.load to get it back into an numpy array.

Knowing when the message has ended

TCP connections are open-ended, which means that the receiving server doesn't know when to stop receiving.

This Python sockets guide puts it nicely:

...you need to realize that there is no EOT on a socket... If the connection has not been broken, you may wait on a recv forever, because the socket will not tell you that there’s nothing more to read (for now). Now if you think about that a bit, you’ll come to realize a fundamental truth of sockets: messages must either be fixed length (yuck), or be delimited (shrug), or indicate how long they are (much better), or end by shutting down the connection.

The simplest way to solve this problem is to simply close the connection after we've sent out the image. But we don't want to close the connection because we want to get a response back from the server (namely, the bounding boxes). So we'll have to specify the message length instead. ^[3] We'll simply prepend the message with the message length. The problem is that the message length itself can also vary, so we'll need to pad it to ensure the first N bytes will be guaranteed to be the header. If you don't pad the message length then you might read part of the array in as your message length.

First we save the numpy array into a BytesIO object using np.save. We can get the message length using message.getbuffer().nbytes, and can pad the message length string using Python's string formatting.

# webcam_stream.py
msg_len = message.getbuffer().nbytes
HEADER_SIZE = 10
b_msg_len = bytes(f"{msg_len:<{HEADER_SIZE}}", 'utf-8')

The :<{HEADER_SIZE} thing right-pads the formatted string with zeros so that it is exactly 10 characters long. Here's an example of what that looks like:

>>> f"{12345}"
'12345'
>>> f"{12345:<10}"
'12345     '
>>> bytes(f"{12345:<10}", 'utf-8')
b'12345     '
>>> bytes(f"{12345:<10}", 'utf-8').decode('utf-8')
'12345     '
# on the server side
>>> int(bytes(f"{12345:<10}", 'utf-8').decode('utf-8'))
12345

On the Docker container we first receive the message length and decode it. (Note that since we are using UTF-8 encoding, a numeric string of length 10 will be 10 bytes long!) We then keep receiving until the number of bytes we've received equals the message length. We then send it to the object detection model (calling the detect function) and send the bounding boxes back.

# ml_model.py
HEADER_SIZE = 10

# First receive the header to know how long the message is:
b_msg_len = connection.recv(HEADER_SIZE)
msg_len = int(b_msg_len.decode("utf-8"))

# Then receive the data in small chunks until we receive msg_len bytes
full_data = BytesIO()
while (full_data.getbuffer().nbytes < msg_len):
    data = connection.recv(4096)
    full_data.write(data)
full_data.seek(0)
print("All data received!")

# Now loading and using the object detection model
img = np.load(full_data, allow_pickle=True)
bboxes = detect(img, 0.3, 0.4)[0]

print("Sending data back to the client")
bbox_bytes = BytesIO()
np.save(bbox_bytes, bboxes)
bbox_bytes.seek(0)
connection.sendall(bbox_bytes.read())

This solution has its limitations: it won't work if we wanted to send multiple images in a single connection. If we sent multiple messages then there would be no guarantee that the first 10 (or whatever) bytes of the message buffer would be the message length. But since we close the socket and re-open a new one after each frame, this is not a problem.

Conclusion

Don't quite understand the implications of this fact, have to do more research ↩︎
From https://beej.us/guide/bgnet/html/ ↩︎
HTTP specifies message length using a "Content Length" or "Transfer Encoding" header field: see this SO link for details. ↩︎

How to install librealsense and pyrealsense2 on the Jetson NX

2020-11-04T00:00:00Z

Introduction

GitHub issue here

I'm going to send in a PR today just to contribute back to the community.

pyrealsense2 is a set of Python bindings for Intel's librealsense library. It is needed to run the code examples showing how to use LibRealSense's Python wrapper.

This set of instructions is a synthesis of both READMEs here: Build LibRealSense from source Build Python Wrappers

Neither of the READMEs give a full set of instructions to follow; each of them is missing some piece. I have thus written this how-to mainly for myself to reference in the future, but also in the hopes that it might save some others' time. These commands have been tested to work on the Jetson NX running Ubuntu 18.04 and using Python 3.6.9.

This how-to provides a complete set of instructions to install librealsense and its bindings on the Jetson NX.

Prerequisites

You should first disconnect your RealSense Camera if it is connected, because the /setup_udev_rules.sh bash script requires your camera to be disconnected.

Instructions

# Installs librealsense and pyrealsense2 on the Jetson NX running Ubuntu 18.04
# and using Python 3
# Tested on a Jetson NX running Ubuntu 18.04 and Python 3.6.9

sudo apt-get update && sudo apt-get -y upgrade
sudo apt-get install -y --no-install-recommends \
    python3 \
    python3-setuptools \
    python3-pip \
	python3-dev

# Install the core packages required to build librealsense libs
sudo apt-get install -y git libssl-dev libusb-1.0-0-dev pkg-config libgtk-3-dev
# Install Distribution-specific packages for Ubuntu 18
sudo apt-get install -y libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev

# Install LibRealSense from source
# We need to build from source because
# the PyPi pip packages are not compatible with Arm processors.
# See link [here](https://github.com/IntelRealSense/librealsense/issues/6964).

# First clone the repository
git clone https://github.com/IntelRealSense/librealsense.git
cd ./librealsense

# Make sure that your RealSense cameras are disconnected at this point
# Run the Intel Realsense permissions script
./scripts/setup_udev_rules.sh

# Now the build
mkdir build && cd build
## Install CMake with Python bindings (that's what the -DBUILD flag is for)
## see link: https://github.com/IntelRealSense/librealsense/tree/master/wrappers/python#building-from-source
cmake ../ -DBUILD_PYTHON_BINDINGS:bool=true
## Recompile and install librealsense binaries
## This is gonna take a while! The -j4 flag means to use 4 cores in parallel
## but you can remove it and simply run `sudo make` instead, which will take longer
sudo make uninstall && sudo make clean && sudo make -j4 && sudo make install

## Export pyrealsense2 to your PYTHONPATH so `import pyrealsense2` works
export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python3.6/pyrealsense2

librealsense and pyrealsense2 should now be installed on your computer. To test this, download an example Python file (e.g. opencv_viewer_example.py) and run it with python3 opencv_viewer_example.py. You should expect to see a window with both RGBA and depth output.

Future extensions

I'm currently trying to build a Dockerfile based on this set of instructions. Let me know if the instructions don't work for you by emailing me at lieu(at)lieuzhenghong(dotcom).

Pair work: the One Weird Trick to finally ship your side projects

2020-10-08T00:00:00Z

Epistemic status: somewhat uncertain.

Introduction

I may have stumbled upon the One Weird Trick for maintaining momentum on side projects. If your side projects are collaborative (most of mine are, these days), the following (which I call pair work) really helps:

Commit to a date and time period (~2h) with your partner.
Get on a call.
One person shares screen. You might use a collaborative editor.
Work together.

This feels so pedestrian that it doesn't seem worth writing about. Maybe everyone already does this but somehow I've missed the memo. In any case---if you haven't been doing this already, I highly recommend trying it.

I want to distinguish pair work from group work. "Group work" often means to portion out a task into smaller subtasks and assign each person a small bit.

Also, I specifically use the term pair work because I don't just mean pair programming--- I believe there are benefits to pair reading-the-documentation, pair understanding, and so on.

I can personally vouch for the effectiveness of this strategy, particularly if you have a project that's been mothballed for quite a while now. Joshua and I meandered around for weeks without any progress until we decided to debug together. We got more done in two hours than we did the past three weeks.

Why and how is pair work so effective? In my experience, there are three main benefits: pair work acts as a strong commitment device, it removes blockers effectively, and shares information amongst all collaborators "for free". It's very effective despite the apparent loss of parallel productivity because the bottleneck in knowledge-based side projects is not raw productivity, but rather motivation, faulty assumptions, and incomplete information.

Pair work acts as a commitment device

This is by far the most important benefit to me. When I start side projects, I always want to work on them but other life stuff gets in the way: be it revising for exams, preparing for a job interview, meeting friends, binge watching YouTube, and the like. Making a public commitment to someone gets you 80% of the way there because you'll be afraid to let them down. But setting a time to work together with a collaborator gets you the last 20%, because without the commitment to work together it's too easy to come up with plausible-sounding reasons to excuse yourself.

Pair work helps remove blockers

Even if you sometimes have the motivation, you might be put off by a tedious part of the project. This is me whenever I'm faced with "weed-clearing" work like setting up builds or debugging import incompatibilities. The problem is that the tedious parts of the project still need to be done, and the more you put it off the more you ruin your productivity. Tedious work is much less so when done together with your collaborator.

Working together also helps when you don't have a clue about how to tackle a particular problem. For instance, I was stuck on a pretty nasty bug when building the board game engine, which blocked me for days as I would dread even trying to solve the problem after banging my head against it fruitlessly for so long.

Talking through a bug with someone else helps a lot; talking through a bug with someone else also involved in the project helps immensely. And obviously two minds are better than one; any problem that appeared insurmountable alone always seems to fall quite quickly.

Pair work also surfaces incorrect assumptions. Most (if not all) misunderstandings arise because one person has a faulty assumption/ mental model. Through the process of argument and discussion that naturally occurs in pair work, these assumptions can quickly be uncovered.

For instance, Joshua is a very experienced data scientist but has little JS experience. He had an incorrect understanding of async and await (specifically, he didn't believe that await blocked execution) and thus he incorrectly rewrote my code. But when working together, I was able to surface this incorrect assumption. This misunderstanding might otherwise have gone unnoticed for a long time.

Pair work puts everyone on the same page

It may be faster initially for two people to split up and work on different things, but this causes two silos of knowledge to be formed. This slows future progress unless significant time is invested familiarising one other with the knowledge of each part. The cost of brain merge-ing may be greater than the gains from parallelisation.

This happened when Joshua read up on the multiplayer server architecture and I did all the client-side work. But that meant that Joshua knew nothing about what I was doing on the client side and what difficulties I was facing, and vice versa for me. This would result in us often talking past each other when discussing how to implement new features. With such a small team and such an interconnected codebase we couldn't not know every part of the system. Had we worked together from the start, initial progress would be slower but overall progress would be faster.

Caveats

This doesn't work without a partner, obviously. There are also some tasks where you do want to work in parallel, like tasks of an exploratory nature e.g. lit review (just to cast a wider net), or trivial tasks where being blocked is not an issue.

You might also get frustrated sometimes if you and your partner disagree on what to do or how to do something.

You could probably do pair work with three people (trio work?), but no more than that. So this would not be feasible with extremely large projects or extremely large teams.

EDIT: Thanks also to Ben Kuhn for pointing out that pair work doesn't work well when there is a big gulf of understanding/knowledge between the partners.

Why this eluded me for the longest time

I had conflated pair work with group work. I hated group work in school. Everyone felt that everyone else was not doing their fair share of the work, or believed that others' work was not up to snuff. It was overall a very grim experience, and I just assumed that would translate to pair work as well: that's just setting N=2, right? But there's a world of a difference between working as a group of two (splitting the work half-half) and working as a pair (doing all the work together).

Now that I've started working as a pair I really can't recommend it enough.

Conclusion

So far, I've found pair working extremely effective: primarily as a commitment device but also with the (significant) side benefits of being able to unblock yourself quicker and having knowledge automatically shared amongst collaborators.

If you do try it out (or have already been practicing it for a while), I'd love to hear from you about your experiences, and if there are any pitfalls to watch out for---shoot me an email.

Home gym ideas

2020-09-27T00:00:00Z

Introduction

Rather than pay $1,000 for a gym membership my neighbour Bryan and I are thinking about building our own home gym.

https://www.tworepcave.com/10581/outdoor-home-gyms-how-to-avoid-rust-or-uv-damage/amp/

https://medium.com/@johngreavesiii/easy-ways-to-protect-outdoor-gym-equipment-961b72727e6c

https://garagegymlab.com/ultimate-guide-budget-home-gym/

Titan Racks:

https://www.titan.fitness/racks/power-racks/t-3-series/?start=0&sz=30

10 Best Budget Gyms:

https://garagegymreviews.com/best-budget-home-gym-setups

How to build a Home Gym on the Cheap

https://www.artofmanliness.com/articles/how-to-build-a-home-gym-on-the-cheap/

Update 4th October 2020

Discussed with Bryan: either put it inside my room or outside in the space between both of our houses.

Prices are actually pretty reasonable: should be <$2000 for the entire set.

Got the link from Marc: directhomegym.sg --- no name Chinese brand that Marc used for his home gym

Update 7th

Update 9th October 2020

I did some research and standing racks are right out because they take up far too much space.

Update Sunday 10th October 2020

Bryan seems to be having cold feet about the whole thing. He balked at the price.

Virtual workspaces --- working in a Minecraft office?

2020-09-26T00:00:00Z

Meta-introduction

I used to write posts on my blog very infrequently because I dreaded the enormous effort and time commitment it needed.

In order to increase my output (rather than constantly having a backlog of posts) I'm going to be trying a slightly new direction: writing shorter posts once I get a germ of an idea, rather than having to have something fully fleshed-out already.

So these posts are going to be incomplete, imperfect, and incorrect even. They may not have a clear thesis or introduction and certainly won't have a clear answer. You can think of them as "scratchpad" posts.

Time taken to write this post: ~4 hours

Introduction

I have recently moved back home, after finishing Oxford, and
I have recently started full-time employment

My sister took my room while I was at Oxford, and so I've had to rethink what sort of room I wanted to have and how to set it up so I could have an effective/ergonomic home office arrangement.

Once I started work I realised that this was only one part of the puzzle. It's one thing to have an ergonomic working arrangement with a standing desk and a split keyboard and external monitors and what not (I have none of those things). But there's another piece of the puzzle: how does one stay motivated while working from home? Let me list some of the issues about working from home that I've experienced.

First is simply that there isn't really a separation between work and play. The only place I can work is in my room, but at the same time the only place I can unwind without distractions is also in my room. And as a i) a software engineer and ii) a sweaty nerd I use the same laptop for both work and play. This problem was most noticeable when I was living alone in the hotel room serving my SHN for two entire weeks.

The fact that there is no separation between work and play matters because it means that either i) I will get distracted during work, or ii) work will bleed into my non-work protected periods.

The second issue about working from home is that of isolation. So I've started working at IMDA for about two weeks now and until now I haven't any clue what my colleagues look like (except my ex-colleagues whom I met during my internship back in Summer 2018). And even though we are supposedly a project "team", I get the feeling that we're just different people working on different things. Also the serendipity is gone. When I was doing my internship I could walk around the office and ask my colleagues what they were working on/ they could give me advice on whatever I was working on. Nowadays, even though someone is technically just only a call away, you won't want to interrupt them. The serendipity and the sense of community is greatly reduced.

Basically, isolation isn't just a mental health thing --- it's also really about camaraderie and the relationship that one has with one's coworkers.

So I've been thinking about these three main questions:

How can I improve the physical ergonomics of my workspace?
How can I improve the mental ergonomics of my workspace?
- how can I prevent myself from being distracted/ prevent work thoughts from spilling into my leisure time?
How can I be less isolated while working from home?

and I don't have the answer, but I thought I'd present the virtual workspace as a possible solution.

The virtual workspace

What do I mean by a virtual workspace? I define it as a spatial, multi-user virtual world where one can

create and move around a customisable space, and
concurrently do your work tasks (check email, write documents, program...)

I believe a good virtual workspace must be spatial and multi-user. Additionally, a good virtual workspace should at least be somewhat structured to encourage work being done in it. I'll explain all of these things shortly.

What some solutions look like now---Spatial VR, Infinite Office

I'd recommend taking a look at Spatial, vSpatial, and Infinite Office.

Here are some random screencaps of Spatial, Spatial VR and Facebook Infinite Office:

Infinite Office has a very slick video with an attractive woman demonstrating how to use the Infinite Office.

I think these solutions are pretty good. They solve the problem of physical ergonomics (somewhat) by allowing you to be productive almost anywhere in your house, with as many monitors as you'd like (the VR headset looked cordless in that very slick Infinite Office video). Spatial gives you a shared virtual office with your coworkers and you can manipulate 3D virtual objects together in that virtual office space. Finally, these solutions solve the problem of "mental ergonomics" with virtual avatars of your colleagues in your "virtual desktop" or "virtual office".

There are two shortcomings with this solution, however. Firstly, they rely heavily on VR to deliver an immersive experience, and most people don't have a VR headset yet (I also don't know whether it's good for your eyes to be constantly looking at the VR screens). And secondly, there's already one group of programs that are very good at creating attractive spaces that people want to spend time in---video games. Could we adapt the video game world into a virtual workspace?

The video game world as virtual workspace

Video games function very nicely as virtual "third places" --- a place separate from home (first place) and the workplace (second place) where people hang out, relax, and banter. I know I've spent hundreds of hours collectively hanging around Henesys or Kerning City in MapleStory, chilling in Habbo Hotel, or skilling and chatting around Burthorpe/Lumbridge in RuneScape. The big question is: could they function as "second places" as well?

I think they could. I think a very nice virtual workplace that got 80% of the way there could be realised using Minecraft in Creative Mode with some mods:

In the morning we log in into the multiplayer Minecraft server and find ourselves in a penthouse suite we built ourselves with a lovely view of the sunrise. (The day-night cycle has been tweaked such that it corresponds to the real day-night cycle of our physical location). We head into a office---a building that looks like an enormous castle from the outside but is actually a modern office inside---by /flying there. There are already people working in the office (these could be your colleagues or other friends/random people), and you can greet them if you so choose. You walk to your office---kitted out to your exact specifications--- and begin work for the day.

First you check your mail. You walk to a special mailbox block and right-click it. This opens up Thunderbird (or your mail client of choice) in a pop-up window, and you respond to the new mails. You note that you have been given several new tasks. You close the mailbox block and your mail client also minimises to the background. You now walk to your enormous whiteboard in your office and add new notes to that whiteboard by right-clicking the whiteboard and typing a note. Looking at the whiteboard, you take stock of the tasks you still need to do and formulate a game plan.

Okay, now it's time to code. You click another custom block and it opens Visual Studio Code (or your IDE of choice). You start working on the first task on the list.

You need to look up some documentation. You right-click a book labelled "MDN Reference" on your bookshelf and the MDN reference opens up within the Minecraft window, left of your VSCode IDE window. You find what you want to find and close the book.

Some time later, there's a ring at the doorbell and you let the person in. It's your colleague who has come to catch up and ask you for help with a technical problem that they're having. You help him work through it with pair programming through the shared IDE extension on VSCode.

After some time, you decide it's time to take a break. In the physical world, you put the kettle on. While the water boils, you leave your (virtual) office and go to the common area where people are milling around. You have some small talk in the common area with the regulars of this office while sipping your tea, asking how their day went and what progress they've made.

Another few hours passes and the sun is setting. You're done for the day. You update the whiteboard, /fly back home, and exit the Minecraft server (or you could continue working on your side projects in the comfort of your own Minecraft home).

Why I think this could be a good idea

What you've written seems like shitty creative writing! What's the point of all this?

Okay, I know this sounds a lot like shitty creative writing. That notwithstanding, I think such a virtual office would have several advantages:

Firstly, the spatial nature of the virtual world enforces a strong separation between work and play. My personal experience is that moving to a different physical location helps me very much in avoiding procrastination / getting into a work mood. I'd often go to OWL to study even though my room was an objectively better place to study (external monitor, larger table..) just because the act of going to OWL got me in the right mood. Ditto with heading to the Inzura office during the weekend to work on the game jam/self-study CS. Another anecdotal ritual: Tak Huen used to put on a nice shirt and tie before sitting at his desk to study for finals---he said it helped him get into the right mindset. In this case, the act of opening the Minecraft server and traveling to the virtual workplace and walking to the corner office is a ritual that tries to imitate the actual ritual of going to the office in the age of Covid-19/without the long commmute. Personally I already try to shutdown my computer and start fresh every day but this ritual just isn't strong enough to get me in the mood sometimes.

This is making my workflow harder on purpose!

or, "Why would I walk around clicking things when I can just Alt-Tab?"

IMO, this is a feature, not a bug. The bottleneck to how much work I get done in a day is not at all how fast I can look up documentation. The bottleneck is my motivation and focus. Notice in the example that individual sites are keyed to books on a bookshelf. This means that you won't be able to easily browse the web or get otherwise distracted if you stay in this Minecraft environment. If you want to check email/WhatsApp/Telegram you'll have to click the correct blocks, and the fact that you have to do that makes that action deliberate rather than a reflex almost.

Why Minecraft?

It doesn't necessarily have to be Minecraft -- it could be any virtual world like Habbo Hotel, Second Life, Runescape, Maplestory, Animal Crossing, WoW, etc... I think Minecraft is particularly attractive because i) you can mod it easily, ii) you have the ability to build your own spaces in a very customisable way. Also, these other games are too "fun"---there's too much to do and I think the temptation to play the game would be too great. Whereas there really isn't much to do in MC Creative Mode.

Why spatiality is important in a virtual workspace

Virtual places with no spatial component exist: I am a avid user of Complice, where there are workrooms you can join that follow a common Pomodoro timer. But I think they're inferior---it's too easy to ignore the workroom. I think humans are intensely spatial creatures and we need to be immersed in an actual space for us to get the full advantages of productivity/connection. Also, the fact that you're ensconced in an actual world where you can only access a curated subset of programs/websites helps prevent procrastination. If you had a nonspatial world it would just be another window you could Alt-Tab away.

Why community is important in a virtual workspace

You could also do this in a single-player world but this wouldn't address the problem of isolation. Just as important as the work itself is the rest and serendipitous chats with coworkers/"co-worlders". At the same time, the fact that everyone has their own private room means that there's a quiet sanctum for you to do deep work without being interrupted.

Examples in the wild

Recurse RC built a very barebones virtual world that ticks a lot of the boxes:

This is Virtual RC. It’s the social space that Recursers enter when they start their days while in batch, and we’ve designed it with the goal of facilitating the kind of serendipitous, synchronous, and ephemeral interactions we think are so important to the experience of doing RC.

The entire world is editable: you can explore, build walls, change colors, write notes, add links, and create audio chats. You can share what you’re working on by leaving a note and see at a glance what’s going on by reading what others have posted. Each of the colorful rooms has a persistent Zoom meeting associated with it, which you enter by clicking on it. The map displays who’s in each room in real time, so you can look for a friend and pop into a room to say hello — much like in the real world!

The calendar blocks display information about scheduled events going on in the rooms, so that when you open Virtual RC and look around, you could see that there’s a machine learning study group going on in Hopper, a video game interest group in Sammet, and a few people working on projects together at the pairing stations. There might also be Recursers hanging out in the virtual kitchen or programming together in the quiet space.

Conclusion

This is a pretty offbeat idea, but I'd like to explore this further. It's certainly not for everybody, but I think there's a subset of people who would really welcome this. Any and all comments are most welcomed.

Acknowledgements

Thanks to CKY for talking to me about this
Thanks to Nick for giving me the link to Recurse RC

Clustering GPS trip data

2020-09-11T00:00:00Z

Introduction

Over the past month we have developed Inzura's data science capabilities by creating a data processing library and formalising a data science workflow. I used the library and workflow to cluster Inzura's corpus of over 2 million anonymised trips.

The general data science workflow I have set out is the following:

Obtain the data
Clean the data
Explore the data
Repeat (try various different approaches to get best results):
- Normalise and reduce the dimensionality of the data
- Perform the clustering on the reduced data
- Inspect the clusters to see if they make sense

Step 1. Obtaining the data

First, I processed the trip data. Recall that a trip is an $T \times 5$ matrix where T is the length of the trip. Because many of the trips have different trip lengths, it's impossible to cluster them in this form. I therefore needed to calculate summary statistics for the trip so I could compare them to each other.

After consultation with Richard and David, I wrote functions to calculate 85 different metrics on the formatted.json data. Some of these metrics include: median absolute velocity, number of stops normalised by trip duration, maximum tangential acceleration, bendiness (the sum of turning degrees per second normalised by trip distance). Importantly, this reduces the dimensionality of any trip from $T \times 5$ to a constant $1 \times 85$ , which makes it much easier to do analysis on.

Once I had come up with the metrics and the functions to calculate them, I had to calculate the metrics for all of the trips. As we had more than 2 million anonymised trips, processing all of them in a reasonable amount of time would have to leverage parallel computation.

To this end, I wrote a multiprocessing library with Richard's help that made use of the Ray library. The library automatically handles the distribution of tasks to processes. Because we didn't want to lose any progress if e.g. the machine failed, the library also saves your progress so you can stop and restart the job anytime, and logs all errors automatically. I have already extensively documented the library and how to use it elsewhere and will not repeat it here.

Processing all the trips on the old bigbeast took several days, despite a ~12x speedup from parallelisation. This really hammers home the fact that we needed parallelisation, or we might still be waiting for the data...

Step 2. Cleaning the data

After all the trips had been processed we were left with a dataset with ~2 million rows and 85 columns. I plotted histograms for each column of the data to visualise the distributions. Doing so allowed me to discover that the data was quite dirty. For instance, there were trips where the average median velocity was something like 1000m/s, which is impossible unless the person was on a rocket. This happens because GPS data can sometimes cut in and out when signal is bad (e.g driving through a tunnel), which leads to an artificially inflated velocity/acceleration/whatever have you score.

I re-plotted the data after removing trips that were patently absurd (average velocity of > 160km/h, maximum acceleration/decceleration of more than 3Gs, trips <30s long) and observed that most metrics followed a power law distribution, with some following a normal distribution. Velocity and acceleration tended to follow a normal distribution while bendiness/stoppiness/trip length followed a power law distribution. This makes sense intuitively.

Here are a few histograms of the filtered data:

Median absolute velocity (filtered)

Idleness (filtered)

Step 3. Exploring the data

To better understand the relationships between the features, I plotted a correlation matrix.

As expected, the different summary statistics of the same underlying data were strongly correlated with one another (median absolute velocity is very highly correlated with 75th percentile velocity, for instance). But there were also some nontrivial correlations. For example, trips that were more "bendy" (measured by the sum of turning degrees normalised by total trip distance) also tended to be more "stoppy" (measured by the number of times a vehicle stops for more than 5 seconds normalised by total trip distance). This makes a lot of sense: This correlation picks up two different trip types: "highway" trips where cars don't stop or bend much at all, versus "urban" trips where cars stop and turn a lot.

Step 4. Reducing and normalising the data

We have 85 different metrics, too many to run clustering on and get results on. This is because clustering algorithms don't work well with high-dimensional points due to something called the curse of dimensionality (but see this CV post for an interesting counterpoint to this).

Essentially, "in high dimensions all examples look alike".

Suppose, for instance, that examples are laid out on a regular grid, and consider a test example $x_t$ . If the grid is d-dimensional, $x_t$ ’s $2d$ nearest examples are all at the same distance from it. So as the dimensionality increases, more and more examples become nearest neighbors of $x_t$ , until the choice of nearest neighbor (and therefore of class) is effectively random

We can fix this by reducing the number of features we consider. I used two different ways: first, by reducing the dimensionality of the data using principal component analysis (PCA), and secondly, by using my domain knowledge of the problem to do manual feature selection.

Principal component analysis (PCA)

My initial approach reduced the dimensionality of the data using PCA. Recall that we have a ~2 million * 85 data set where many features are very highly correlated with one another. It's likely that we would be able to explain a large proportion of the variance without using all of the features.

PCA is sensitive to scale so I first had to normalise the data. After removing absurd data, I used scikit-learn's MinMaxScaler to scale all the metrics to lie between 0 and 1. Then I ran PCA to generate a "scree plot" (shown below), which shows the proportion of variance explained by each principal component. I found that just three components were sufficient to explain more than 50% of the variance, and 37 components were sufficient to explain more than 99% of the variance.

Manual feature selection

While the factor analysis was successful, I worried that the components derived by PCA (and the resulting clusters) would be uninterpretable. I therefore also did manual feature selection, selecting the following six values:

Median absolute velocity
Number of stops normalised by trip duration
Percentage of GPS points that were snapped to a road link
Max radial acceleration
Max positive tangential acceleration
Idleness

I chose these features because I thought they would be the most informative. While I could have chosen more features that were also likely to be informative (e.g. bendiness and maximum negative tangential acceleration), the more features one chosses, the more likely one will run into the curse of dimensionality, and the poorer the clusters will be.

I did not remove outliers when doing manual feature selection, instead choosing to take the $log(x+1)$ transform:

normalised_df = filtered_df.apply(np.log1p)

I plotted each metric's distribution after taking the log x+1 transform and the MinMaxScaler to standardise between 0 and 1. The data was surprisingly well-behaved. Both radial and tangential acceleration seemed to conform to a normal distribution, while number of stops and percentage snapped followed a power law. Absolute velocity seemed to follow some sort of truncated normal distribution, which makes sense because speed limits exist (both legal and physical). Here are the six histograms:

Median absolute velocity

Idleness

Number of stops, normalised by trip distance

Maximum radial acceleration

Maximum positive tangential acceleration

Percentage of snapped road links

Step 4. Perform clustering on the normalised and reduced data

After several false starts, and in keeping with the KISS principle, I decided to use the simplest clustering algorithm possible---k-means clustering. The main problem with K-means clustering is that one must specify the number of clusters. After discussion with Richard and David, we decided that the a priori number of clusters should be six (walk, car, bus, train, plane, ferry), and so I used that number ^[1]. As usual, scikit-learn provides a one-liner:

kmeans = KMeans(n_clusters=6, random_state=0).fit(y)

After clustering the dataset, we need to visulise our clusters. This is not a trivial task as our trips are in 6D space, and I know of no good way to visualise high dimensional clusters. Instead, I took all possible pairs of the six metrics and plotted each of them as a 2D scatterplot, which is much easier to interpret. There are $6 * 5 = 30$ scatterplots in total (each metric has a scatterplot with all the other metrics), but half of them are mirror images, so this leaves 15 total scatterplots. I used matplotlib to create a lower triangular matrix of scatterplots, and the results are as follows:

Although the actual clustering only took one line of code, there was in reality a significant amount of work that went into it. I ran into several dead ends that initially seemed promising, and I list them here.

Things that didn't work

Hierarchical clustering --- too slow

I first tried to use hierarchical clustering so that we didn't have to specify the number of clusters but this didn't work because hierarchical clustering runs in O(N^3) time, which is too slow for the number of data points we have. In fact, SciPy ran into a memory error when trying to do heirarchical clustering on just 1/10th of our dataset.

PCA and then K-means clustering --- too uninterpretable

As mentioned, I used PCA to reduce the dimension of the 85 or so calculated metrics and then performed K-means clustering on those reduced dimensions. While this seemed to work OK, it wasn't really that interpretable. I think this approach is probably the right way to go in the future. Right now, though, I would need to have a better understanding of how different types of trips look like so that I can manually inspect the clusters.

Step 5. Perform data analysis on those clusters

We've visualised the clusters. Now let's take a closer look at them and see what we can learn.

The first thing we notice is that there are outlier trips with high absolute velocity. This is by design as I wanted to cluster all of these outlier trips together with plane trips which also have a high absolute velocity. We also see in the second plot from the top left that there are a few outliers with high absolute velocity and almost 0 snapped percentage: these are likely either plane trips or trips with poor GPS data.

The second thing we notice is that the clusters don't seem all that great. While we can definitely see distinct clusters, the clusters seem to have significant overlaps. This could be because there are too many points to visualise, because 6D space doesn't map cleanly onto 2D space, or because the clusters themselves just aren't that great (more on this later).

The third thing to notice is that there seems to be a negative correlation between number of stops and maximum acceleration, as well as a negative correlation between number of stops and median velocity. You'll see this most clearly when looking at the second plot in the third row, num_stops_value (Y) against rad_accel_max(X). This is as we expect. The negative correlation between number of stops and maximum could be because cars making trips that stop more tend to be urban trips (slower) while cars making trips that stop less tend to be highway trips (faster). Or it could also be reflecting a difference in trip modality: buses tend to stop a lot and tend also to be slower/accelerate slower.

In order to know more about the clusters, I plotted the number of trips belonging to each cluster in the figure below.

We can see that the clusters are unbalanced, which is a good thing, because the vast majority of trips in our dataset are car trips. However, they are not unbalanced enough ---they don't match our priors on how likely each type of cluster is likely to be. Of course, it would be quite unrealistic to expect the clusters k-means identified to coincide entirely with the clusters we care about.

Why didn't K-means give good clusters?

We've used the simplest possible clustering algorithm (a veritable workhorse), K-means clustering. However, there are many shortcomings with K-means clustering, which could be the reason why the clusters are not as good as we hoped.

K-means clustering may not do well with very uneven clusters

For instance, k-means may sometimes fail with uneven clusters. The following image shows an example where k-means fails to cluster correctly:

The reason is because the top-right cluster is very big and the top-left cluster is very small. So the k-means algorithm would rather forsake the loss of the small number of points at the top-left and use the "freed-up" cluster to decrease the loss in the big cluster.

I suspect that this is happening in our case. We have a tiny number of outliers (e.g. plane trips with high average velocity, low percentage snapped, etc.), and this should be in a cluster of its own, but K-means will not do that if the number of outliers is too small.

K-means clustering cannot deal with intracluster covariance

Two other shortcomings with K-means clustering are laid out very nicely in the following post.

The first shortcoming is that K-means clustering can't account for covariance.

One way to think about the k-means model is that it places a circle (or, in higher dimensions, a hyper-sphere) at the center of each cluster, with a radius defined by the most distant point in the cluster.

K-means doesn't work well if the clusters have some positive or negative covariance (such that the shape of each cluster is non-circular).

K-means tries to fit all points to a cluster even if they may be outliers

The second and more important difference is as follows:

Ideally, a clustering algorithm should be able to label points as outliers and not force-fit them into an existing cluster. While Gaussian mixtures can label points as not belonging to any cluster at all, K-means cannot do this, which means outliers will skew our cluster centers and thus result in incorrect results.

It would be helpful to have some sort of "confidence" in whether or not a particular trip belongs to any cluster at all, and if the probability is too low, label that point an outlier. Gaussian models do this:

The second difference between k-means and Gaussian mixture models is that the former performs hard classification whereas the latter performs soft classification. In other words, k-means tells us what data point belong to which cluster but won’t provide us with the probabilities that a given data point belongs to each of the possible clusters.

Ideas for future work

Using the fact that we have a corpus of guaranteed car trips for anomaly detection

We know that the vast majority of recorded trips (> 99%) are car trips. And we also have what is in fact a "ground truth" with the beacon started trips from company 5 and 11 (these are trips that are guaranteed to be car trips).

We should make use of this prior knowledge somehow! Our current unsupervised learning model doesn't make use of any of this information, which is a real waste.

We can do what is in fact a semi-supervised learning model (anomaly detection). In essence, this will flag any trips that are too different from the beacon started trips, be it walk/ferry/train/plane. These trips can then be flagged out and manually inspected, or some sort of model can be trained on them.

I believe that we can use Gaussian mixture models to implement this: see this article for a nonrigorous overview. A big advantage of using Gaussian mixture models is that we can explicitly set a priori prior probabilities of how likely any random trip will belong to a particular cluster.

Running PCA within clusters to compare the difference between intracluster and intercluster variance

Finally, here's an idea I got from a friend:

Do the clustering, subsample to a cluster and redo PCA on that cluster. That'll show how variance is differently explained between clusters and within a cluster.

For instance, if you had two clusters, and the PCA found that the loading on absolute velocity was high, and then you dialed into one cluster and ran PCA there, and the PCA there found that the loading on number of stops was high, then this would suggest that the variance between clusters was more due to abs velocity (e.g. plane vs non-plane) and the variance within a cluster was more due to number of stops e.g. buses stop a lot and cars tend not to -- but abs velocity doesnt vary as much between the two modes of transport.

Conclusion

Our contribution is not just the results of the data analysis, but also the process that I have outlined here. With the library and the workflow established, future data science tasks will be more streamlined. We will be able to process any data, get more insights from it and evaluate the results very quickly in the future.

One possible method if the number of clusters was unknown a priori would be to plot the Bayesian and Aikeke Information Criteria (BIC, AIC), and just eyeball when the BIC plateaus. ↩︎

Board Game Engine

2020-09-01T00:00:00Z

Introduction

I've put this project on hold for now. GitHub repo here

System architecture

ClientMain.ts:

Client Main initialises

ClientGameCore.ts
- GameState
UIHandler.ts

ClientMain.js has an init function that fetches the two JSON files (game_state.json and image_map.json) and initialises ClientGameCore and UIHandler (handles clicks).

The ClientGameCore inherits GameCore and in turn initialises GameState.

GameCore.ts

  // Idea: store a finite state machine here so that we can be sensitive
  // to context.

  // There are a couple of actions I want to support right now
  // as an MVP:

  // == OPTION TYPE 1 ==
  // Right-click on object to open context menu, then
  // left-click on an option.

  // Choice of options:
  // --> change state of Entity (maybe this is a submenu)
  // --> enter "change of Zone" mode where the next left click on a new Zone
  // changes the entity to that zone, if permitted
  // --> enter "change position" mode where the next left click at any position
  // moves the entity to that position

  // We should also implement the following option type:
  //
  // == OPTION TYPE 2 ==

  // Left-click an object to enter "drag" mode, then
  // move the mouse anywhere to move the entity's position.
  // Left-click again to drop the entity.
  // Note that this doesn't change the entity's zone.

  //             Change State Mode
  //                    ^
  //                    |
  //                    |
  // Base Mode --> Entity UI Mode -->  Change Position Mode
  //   |  ^             |
  //   |  |             |
  //   v  |             v
  // Drag mode    Change Zone mode

  // We will also have a ContextMenu that is generated
  // and probably also some flags like CurrentlyDraggingEntity
  // and EntityCurrentlyDragged

  //
  //                                ________________                _________
  //                                |               |               |        |
  // UI [posx, posy, LMB, RMB] ===> |    GameCore   | Action[] ===> | Server |
  //                                |_______________|          <=== |________|
  //                                        |
  //                                        | Action[]
  //                                        v
  //                                    GameState
  //
  /*
  // == Time-based loop (every 15ms?): ==
  // Client Core sends actions to Server Core from the action queue
  // Server Core accumulates and validates actions from all Client Cores
  // Server Core sends actions to Client Cores
  // Client Core applies actions to GameState
  // Client Core renders GameState

  // == Event-based loop: ==
  // on RightClick on Entity, Client Core generates Context UI
  // ClientCore responds to clicks on Context UI and generates ServerActions

  // on MouseMove or any other Click, Client UI sends mouse positions/clicks to Client Core
  // Client Core converts mouse states to ServerActions 
  // Both types of actions are pushed into an action queue

Building a Python parallel processing pipeline package (R-3PO)

2020-08-29T00:00:00Z

Introduction

Richard Jelbert (CEO of Inzura) and I whipped up a Python parallel processing pipeline package (just rolls off the tongue, doesn't it) over a couple of days. We used it to process a large number (over 3 million) of JSON files.

Richard's Parallel Processing Protocol, or R-3PO for short (like C-3PO) is an embarassingly trivial package built on top of Ray to make embarassingly parallel problems embarassingly easy.

Use case: Suppose you have lots of data files that need to be processed in the exact same way with the same function. And suppose you want to save the results of that processing into a CSV file. This is an embarassingly parallel problem: it should be easy.

Embarassingly easy, even: R3PO lets you do it with a config.yaml file and two lines of code. It automatically handles the distribution of tasks to processes, saves your progress so you can stop and restart the job anytime, and logs all errors automatically.

This package is probably useless to you unless your workflow and use case are very similar to mine. But if it is, I believe it could save you an hour or two from needing to write boilerplate code.

1 minute guide

Have the following two files in your directory.

config.yaml:

job_name: count_produce
processes: 2
source_path: /home/lieu/produce_log
source_file_part: .json
working_dir: /home/lieu/working_dir
output_path: /home/lieu/output_dir

main.py:

# main.py

from r3po import jobbuilder, jobrunner
from count_fruits import count_fruits

# Build jobs
build_jobs('./config.yaml')

# Now run them
run_jobs('./config.yaml', count_fruits)

Running python3 main.py will run the count_fruits function on all .json files in /home/lieu/produce_log/ and log the results in numbered csv files inside /home/lieu/output_dir/.

Sounds like something you'd find useful? Read on...

Quickstart guide

We'll work though a very simple example available on the Github repo. First, let's clone the repo and go to the sample directory:

Cloning the repo

git clone https://github.com/lieuzhenghong/r3po
cd r3po/sample

The directory will look like this:

./r3po/sample/
├── produce_log/
    ├── file_0.json
    ├── file_1.json
    ├── file_2.json
    ├── file_3.json
    ├── file_4.json
    ├── file_5.json
├── config.yaml
├── count_fruits.py
├── main.py

Setting the scene

Imagine you're the king of Cornucopia and you own one bajillion acres of land. You've allocated an acre of land to each one of your loyal subjects for them to farm as they see fit.

One day, you decide that you'd like to know how much food is produced in Cornucopia. So you ask your subjects to report their yields. Your subjects are well-acquainted with both farming and the JSON spec, so let's take a look at file_0.json:

{
  "produce": {
    "fruits": {
      "apples": 24,
      "bananas": 33,
      "cantaloupes": 3
    },
    "vegetables": {
      "carrots": 15,
      "cucumbers": 22,
      "cabbages": 21
    },
    "meat": {
      "beef": 12,
      "pork": 2
    }
  }
}

Now as the king you're not that interested in the various types of fruits per se---perhaps you only want the total sum of each type of produce. Let's say you want something like the following:

{
    "fruits": 60,
    "vegetables": 58,
    "meat": 14,
}

So you write a simple function that does exactly that. The details are not important, but the key is that any such function must take in a filepath string and return a dictionary for R3PO to work with it.

# count_fruits.py
import json

def count_fruits(produce_log_filepath):
    with open(produce_log_filepath, 'r') as f:
        produce_counts = {}
        data = dict(json.load(f))
        all_produce = data['produce']
        for produce_type in all_produce:
            produce_counts[produce_type] = 0
            for produce in all_produce[produce_type]:
                produce_counts[produce_type] += int(
                    all_produce[produce_type][produce])
        return produce_counts

So you'd want to run this function for every file. However, you have a bajillion acres of land and thus a bajillion JSON files (instead of the six files in the sample folder) and so we'd like to speed it up by running the code in parallel. Let's see how to do it with the R-3PO pipeline.

First, let's open config.yaml and take a look at the file.

job_name: count_produce
processes: 2
source_path: /home/lieu/produce_log
source_file_part: .json
working_dir: /home/lieu/working_dir
output_path: /home/lieu/output_dir

Most of these are pretty self-explanatory (more details later), but basically this means that

R3PO will spin up two parallel processes,
look for any file ending with .json in /home/lieu/produce_log/, and
write the results in /home/lieu/output_dir.

Now let's open the main.py file and see what's in it (a whopping two lines of code):

# main.py

from r3po import jobbuilder, jobrunner
from count_fruits import count_fruits

# Build jobs
build_jobs('./config.yaml')

# Now run them
run_jobs('./config.yaml', count_fruits)

Not much to say about this. Let's run it and see what happens:

Running the code

# bash pip3 install r3po
# python3 main.py

Building job called: count_produce
Reading job source directory and generating master job CSV file...
Scanning for files ending with .json
Pending files generated.
Removing old node jobfiles...
Writing new node jobfiles...
Number of files: 6
Number of pending files: 6
2020-08-30 16:38:05,916 INFO resource_spec.py:223 -- Starting Ray with 5.47 GiB memory available for workers and up to 2.73 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-08-30 16:38:06,302 INFO services.py:1191 -- View the Ray dashboard at localhost:8265
Spinning up Ray processes...
Process 0 spawned!
Process 1 spawned!
None
None
(pid=336291) Processed trip /home/lieu/dev/r3po/sample/output_dir/0.results.csv in node 0.
(pid=336291) Processed trip /home/lieu/dev/r3po/sample/output_dir/0.results.csv in node 0.
(pid=336291) Processed trip /home/lieu/dev/r3po/sample/output_dir/0.results.csv in node 0.
(pid=336283) Processed trip /home/lieu/dev/r3po/sample/output_dir/1.results.csv in node 1.
(pid=336283) Processed trip /home/lieu/dev/r3po/sample/output_dir/1.results.csv in node 1.
(pid=336283) Processed trip /home/lieu/dev/r3po/sample/output_dir/1.results.csv in node 1.

Okay, everything looks good! Let's take a look at what has happened. There were a lot of print statements which are self-explanatory.

We can see that Ray has spun up two parallel processes (exactly what we specified in the config.yaml file). Additionally, we can see that R3PO has created an output_dir with two results.csv files.

We also have a working_dir/ with done.job and nodejobfile.txt files, but we won't worry about that for now.

./r3po/sample/
├── output_dir/
    ├── 0.results.csv
    ├── 1.results.csv
├── produce_log/
    ├── file_0.json
    ├── file_1.json
    ├── file_2.json
    ├── file_3.json
    ├── file_4.json
    ├── file_5.json
├── working_dir/
    └── tracking/
        └── ..../file_0.json
            └── done.job
        └── ..../file_1.json
            └── done.job
        └── ..../file_2.json
            └── done.job
        └── ..../file_3.json
            └── done.job
        └── ..../file_4.json
            └── done.job
        └── ..../file_5.json
            └── done.job
    ├── 0.nodejobfile.txt
    ├── 1.nodejobfile.txt
├── config.yaml
├── count_fruits.py
├── main.py

Opening up the two results.csv files, we can see that the function has indeed been calculated for all of the input JSON files. Because we spun up two processes, there are two results.csv files:

# 0.results.csv
fruits,vegetables,meat
60,58,14
3939,58,14
6456,3743,14

# 1.results.csv
fruits,vegetables,meat
21,1539,79
60,58,14
5666,15,15

And we're done! These CSVs can then be converted and concatenated with pandas's pd.concat function to get pandas DataFrames, which is the format in which I do the majority of my data analysis.

How it works in more detail

Overall abstraction

This pipeline runs a function f on a large number of input files in parallel and logs the results into a CSV.

It is made up of two files. The first file jobbuilder.py reads the source_path from config.yaml. It looks for all files inside source_path that match a user-specified criteria. It writes these file paths inside nodejobfile text files that tell each process which files to work on.

The second file jobrunner.py spins up the processes. Each process looks at its nodejobfile text file, and works through the list.

For each $FILEPATH listed in nodejobfile:

the process opens the file,
runs some function f on that file,
and (tries to) append the output to a .csv file.

If the function successfully runs, an empty file called done.job is created in <$WORKING_DIR>/tracking/<$FILEPATH>/as a record of completion. If the function throws an exception, then a file called error.job is created instead.

Keeping a record of file completion means that the processes can be terminated and restarted at any time without going through the same files again.

Limitations and requirements

The pipeline requires the following to work:

A config.yaml file that follows a particular specification
A function that follows a specific contract (see next section)
All your input files must end with the same suffix.

The function contract

The function you call must take as input an absolute filepath to the file.
It must return a Dictionary that will be passed to csv.DictWriter.
Furthermore, every Dictionary object returned must have the same keys. If it is not able to return such a Dictionary, it must raise an Exception.

The `config.yaml` file

This is what a config.yaml file should looks like. All fields are required.

job_name: generate-trip-visualisations
output_path: /home/lieu/RJ/trip_processing_research/.generate-trip-visualisations
processes: 12
source_file_part: formatted.lz4
source_path: /media/lieu/bigdata/anonymous/rjrun
working_dir: /home/lieu/RJ/trip_processing_research/.generate-trip-visualisations

count_all_files and count_job_files are added by the program and list the number of files in the job and the number of files remaining in the job respectively.

job_name is the name of the processing job. It is used in print statements only.

processes is the number of parallel processes to spin up.

source_path is a filepath string pointing to the folder where the source files are. source_file_part is used for filepath matching: R3PO will traverse all subfolders inside source_path and find all the .json files. In this case,

working_dir is a filepath string that points to a folder to store intermediate files. where the nodejobtxts will be, as well as the done.job and error.job files.

output_path is a filepath string that points to a folder where you want the results of the computations to be written to CSV files. If the folder does not exist, one will be automatically created.

What jobbuilder.py does

The jobbuilder module is responsible for getting the filepaths of all the input files and writing these filepaths to a CSV that jobrunner can read later. It does not write the filepath if the file has already been processed or if an error occurred whilst processing that file.

This is how it does it:

Read config.yaml file and find source_path, working_dir, and source_file_part
Create the folder working_dir if one does not exist
Look for file paths that match source_path/**/*.source_file_part, and do not already have a done.job or error.job
Write X.nodejobfile.txt by dividing the file paths up equally across all X process nodejobfiles

Why does this even exist?

Admittedly, the code is pretty trivial, and quite overfitted to my specific use-case. Why did I even bother packaging it up in the first place? I offer three reasons:

If you think about it, the code is not that trivial. It's easy enough to parallelise code in Python with the multiprocessing library. But it can be tricky to write code that allows jobs to terminate and restart arbitrarily, while not reprocessing already-processed files.
It follows from point 1 that this package will save you spending an hour or two reinventing the wheel if your use-case and dataflow are similar to mine.
I was lazy and wanted to be able to pip install rather than git clone-ing and moving the files around manually.

Conclusion

Thanks for reading!

GSoC 2020: implementing real-time metrics for MGGG's Districtr app

2020-08-28T00:00:00Z

Introduction

For Google Summer of Code 2020, I implemented a new feature to MGGG's flagship webapp Districtr. Districtr has an existing feature where you can "colour in" districts using a brush tool: each colour represents a different district.

My contribution can be seen in the bottom right corner of the below GIF. As you draw the districts with the brush, three metrics update in real time:

the contiguity status (whether districts drawn are one continuous whole or get broken up in the middle);
The number of cut edges;
How the number of cut edges compares to a sample of plans generated by the Recom redistricting algorithm

Because I know of no way to find contiguity/number of cut edges of a graph partition in-browser, this thus requires server-side computation. In this post, I explain how these metrics are calculated, how I deployed the server that does the calculations, and how I built the front-end to send requests and display the results.

The main PR is #212 where I implement cut edges here (PR #212) but I also implemented contiguity before that here (PR #209)

Understanding the problem

A state can be thought of as a dual graph, and a particular district assignment of precincts to districts is in effect creating a partition of that dual graph. Given this partition of the dual graph we can then calculate out whether or not the subgraphs are contiguous and how many cut edges there are in the partition.

How should we calculate this? I determined that the best way to do so was to use the GerryChain library (written in Python). The biggest reason is dogfooding: since GerryChain is being actively developed by the MGGG group, it made sense to use GerryChain rather than some other library.

Software infrastructure in brief

This is a very simple client-server model.

The Districtr webapp is built on lit-HTML. The histogram is dynamically drawn with Vega.

The server is a Flask server that calls GerryChain code.

The solution in brief

Upon a brush stroke, the client sends a POST request to the Flask server (hosted on PythonAnywhere).
The server calculates the metrics and sends them as a response to the client.
The client takes the response and rerenders the DOM appropriately.

Step 1: The client sends the district assignment to the server

Drawing districts with the brush in the Districtr website updates an assignment object. This object looks something like this (but see Appendix):

The key is the voting precinct ID and the value is the district it has been assigned to. For instance, precinct 19001 has been assigned to district 2.

This assignment object is sent to the Python server for computation.

Step 2: The server calculates metrics and responds to the client

The server receives the assignment object, and converts it to a GerryChain Partition object provided it has that state's dual graph:

partition = gerrychain.Partition(state_graph, assignment, None)

Using the Partition object we calculate cut edges and contiguity metrics, then send them to the client.

Step 3: the client updates its UI with the server's response

The client updates its UI appropriately with the server's response. The histograms are rendered with Vega (more on this later).

At a high level, that's all there is to it! I'll now explain the solution in detail.

The solution in detail

Step 1: The client sends the district assignment to the server

The assignment object is not just sent once, but constantly as the user draws districts on the Districtr app. In the ToolsEditor() function in /plugins/tools_plugin_js there is a brush.on event that detects any change in district assignment, and sends the new assignment to a ContiguityChecker function. The spatial_abilities() condition checks if contiguity checks have been enabled for that particular state.

(TODO: this function enables/disables contiguity and cut edges all at once. We should find some way to fix this, or at least rename it. )

const c_checker = ContiguityChecker(state, brush);

brush.on("colorop", (isUndoRedo, colorsAffected) => {
  savePlanToStorage(state.serialize());
  if (spatial_abilities(state.place.id).contiguity) {
    c_checker(state, colorsAffected);
  }

  if (planNumbers) {
    planNumbers.update(state, colorsAffected);
  }
});

The ContiguityChecker function lives in src/map/contiguity.js, where it sends the district assignment to the PythonAnywhere server through a POST request:

export default function ContiguityChecker(state) {
  let saveplan = state.serialize();
  console.log(saveplan);
  const GERRYCHAIN_URL = "//mggg.pythonanywhere.com";

  fetch(GERRYCHAIN_URL, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify(saveplan),
  });
  ... // code removed for brevity ...
}

Step 2: The server calculates metrics and responds to the client

The Python server is as simple as can be. It is a Flask server with only one route:

@app.route('/', methods=['POST'])
# Takes a Districtr JSON and returns whether or not it's contiguous and number of cut edges.
def plan_metrics():
    plan = request.get_json()
    ... # code removed for brevity
    districtr_assignment = plan["assignment"]
    ... # code removed for brevity

As we can see, the server just parses the POST request and then pulls out the Districtr assignment object. Now what do we do with it?

The first step is to convert the Districtr assignment object we've received to a format that GerryChain can work with. There's already a function that does something similar, but it i) loads from a file instead of an object, and ii) doesn't handle incomplete assignments (i.e. when the user doesn't fill up the map completely). I therefore wrote my own code that could handle both of those things.

(TODO: make a PR to GerryChain to add new classmethod)

How we handle incomplete assignments is somewhat arbitrary, but I decided to put all non-assigned nodes into a district of their own (district -1). After the assignment is complete, we then use the gerrychain.Partition function to create a partition and calculate cut edges and contiguity.

partition = gerrychain.Partition(state_graph, assignment, None)

# Now that we have the partition, calculate all the different metrics

# Calculate cut edges
cut_edges = (partition['cut_edges'])

# Split districts
# TODO timeit this --- how long does it take?
split_districts = []
for part in gerrychain.constraints.contiguity.affected_parts(partition):
    if part != -1:
        part_contiguous = nx.is_connected(partition.subgraphs[part])
        if not part_contiguous:
            split_districts.append(part)

# Contiguity
contiguity = (len(split_districts) == 0)

We send the calculated metrics back to the client:

response = flask.jsonify({'cut_edges': len(cut_edges), 'contiguity': contiguity, 'split': split_districts})

Step 3: the client updates its UI with the server's response

The fetch request the client sends has hopefully returned with no error. If so, we start updating the UI and do three things:

set contiguity status,
set number of cut edges,
draw a histogram of cut edges and show where the current plan lies.

    fetch(GERRYCHAIN_URL, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify(saveplan),
    })
      .then((res) => res.json())
      .catch((e) => console.error(e))
      .then((data) => {
        console.log(data);

        // Set contiguity status
        setContiguityStatus(data);

        // Set number of cut edges
        let num_cut_edges = setNumCutEdges(data, state_name);

        // Embed the canvas
        const state_name = saveplan['placeId'];
        const container = document.querySelector(".cut_edges_distribution_vega");
        const vega_json = __form_vega_json__(state_name)
        vegaEmbed(container, vega_json).then((v) => {

          // Wait for vegaEmbed to generate the canvas, then draw a line on it
          const canvas = document.querySelector(".marks");
          const left_bound = state_info[state_name]["lr_bounds"][0];
          const right_bound = state_info[state_name]["lr_bounds"][1];

          __draw_line_on_canvas__(canvas, num_cut_edges, left_bound, right_bound)
        })

The setContiguityStatus and setNumCutEdges are pretty straightforward functions where I just change the innerText of the relevant HTML elements. If we were doing this in React it would probably be better to store contiguityStatus and numCutEdges as state and have some sort of hook to update, but this approach works fine:

function setContiguityStatus(contiguity_object) {
  let contiguous = contiguity_object.contiguity;
  document.querySelector("#contiguity-status").innerText = !contiguous
    ? "Districts may have contiguity gaps"
    : "Any districts are contiguous";
}

The most complicated function is rendering the histogram of districting plans and drawing the red line on the histogram, and I'll go through that now.

Generating the distribution of cut edges

First of all, what is the histogram representing? Where does it come from? The histogram shows the number of cut edges from 10,000 sample districting plans. These plans were generated using the Recom redistricting algorithm, a districting algorithm developed by MGGG. The maths is complicated, but essentially it's a Markov Chain technique; it starts from some random assignment, and to generate a new assignment it changes the assignment of certain districts in a "smart" way so as to maintain contiguity, population balance and relative compactness. Each one of these generated plans has a number of cut edges. After running the algorithm for 10,000 steps, we'll have gotten a distribution of the number of cut edges, and this is exactly what we are plotting.

How districting plans are generated by the Recom algorithm. Image from the Recom paper

For each state we have precomputed 10,000 sample districting plans and stored it in a csv file. This process takes anywhere from a few hours to a few days, but only has to be done once: once the distribution of cut edges is known it can simply be used again and again. The csv files live in assets/cut_edges_histograms/$STATE_chain_10000/csv.

Generating the histogram

The next step is drawing the histogram on the UI. I initially used a PNG generated by matplotlib.hist, but this turned out to be nonideal because the UI could resize and PNGs are not responsive. We therefore need to generate histograms dynamically. and after some searching I decided on Vega-Lite.

Vega-Lite is a "high-level grammar of interactive graphics" that "provides a concise, declarative JSON syntax to create an expressive range of visualizations for data analysis and presentation". To create the histogram I needed to write a JSON object that specified what data to plot and how I wanted it to be plotted:

// Returns the Vega Lite schema needed for rendering the state cut edge histogram dynamically
function __form_vega_json__(state_name) {
  const vega_json = {
    $schema: "https://vega.github.io/schema/vega-lite/v4.json",
    data: { url: state_info[state_name]["state_csv"] },
    mark: "bar",
    width: "container",
    autosize: {
      type: "fit",
      contains: "padding",
    },
    encoding: {
      x: {
        bin: { maxbins: 40 },
        field: "cut_edges",
      },
      y: { aggregate: "count" },
    },
    config: {
      axisY: {
        labels: false,
        title: null,
      },
      axisX: {
        title: null,
      },
    },
  };

  return vega_json;
}

Then we use a helper function vegaEmbed to embed the visualisation into the DOM. VegaEmbed takes a container element and the vega_json schema and generates a HTMLCanvasElement with <canvas class='marks'> within the container element:

    vegaEmbed(container, vega_json).then((v) => {
        // here v is the vega view, but we will ignore it
        const canvas = document.querySelector(".marks");
        const left_bound = state_info[state_name]["lr_bounds"][0];
        const right_bound = state_info[state_name]["lr_bounds"][1];
        __draw_line_on_canvas__(canvas, num_cut_edges, left_bound, right_bound)
    }

I take that canvas element and pass it to my __draw_line_on_canvas__ function, which draws a line that represents how the current districting plan compares to the sample of plans.

You may be wondering: What is the state_info object? It is basically a map that tells the code where to find the data CSV object, and the upper and lower bounds of the number of cut edges in the 10,000 generated plans. We need the upper and lower bounds in __draw_line_on_canvas__ to draw the line, as you'll see later.

const state_info = {
  connecticut: {
    state_csv: "../assets/cut_edges_histograms/CT_chain_10000.csv",
    lr_bounds: [110, 270],
  },
  iowa: {
    state_csv: "../assets/cut_edges_histograms/IA_chain_10000.csv",
    lr_bounds: [32, 60],
  },
  texas: {
    state_csv: "../assets/cut_edges_histograms/TX_chain_10000.csv",
    lr_bounds: [2720, 3440],
  },
};

Drawing the red line on the histogram

The red line in the image below shows how the current plan compares to a set of generated plans in terms of number of cut edges. The main difficulty is that we have a number of cut edges---but to draw the line, we need to convert that to an x-position on the histogram.

To do that, we need the upper and lower bounds of the cut edges distribution:

If the number of cut edges is below the lower bound, we set line position to 0.
If it is above the upper bound, we set it to the canvas width.
And if it lies somewhere in between, then since the x-axis is uniform, the x-position given a number of cut edges n is

xPosition = w * \frac{(n - leftBound)}{(rightBound - leftBound)}.

In this example, suppose the canvas is 300px wide, and the lower bound and upper bound are 32 and 60 cut edges respectively. Then 39 cut edges would correspond to

300px * \frac{39 - 32}{60-32} = 75px .

We then get the canvas's Canvas2DContext and draw a line at the correct x-position, which in this case is 75px. The code looks like this:

function __draw_line_on_canvas__(
  canvas,
  num_cut_edges,
  left_bound,
  right_bound
) {
  const w = canvas.width;
  const h = canvas.height;
  let line_position = 0; // this is going to be a value between 0 and w

  if (num_cut_edges <= left_bound) {
    line_position = 0;
  } else if (num_cut_edges >= right_bound) {
    line_position = w;
  } else {
    line_position =
      ((num_cut_edges - left_bound) / (right_bound - left_bound)) * w;
  }

  console.log("Line position: ", line_position);

  console.log("Line position:", line_position);
  const ctx = canvas.getContext("2d");
  ctx.lineWidth = 2;
  ctx.strokeStyle = "red";
  ctx.beginPath();
  ctx.moveTo(line_position, 0);
  ctx.lineTo(line_position, h);
  ctx.closePath();
  ctx.stroke();
}

And that's it!

What I learned

How to work with a complex existing codebase

In my previous projects I have always started writing code from scratch, and that's much easier because code complexity is lower, you know where everything is, you understand every single part of what you're building, there are no complicated build pipelines, and you can always just push to origin/master.

This was significantly harder because there was already an existing medium-sized codebase (~10,000 LOC) running tech stacks/folder structure/build pipelines unfamiliar to me.

lieu@pop-os ~/d/districtr (lieu-branch)> cloc src/ sass/ html/
     158 text files.
     158 unique files.
       0 files ignored.

github.com/AlDanial/cloc v 1.82  T=0.05 s (3398.9 files/s, 317945.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JavaScript                     110            703            543           9030
Sass                            35            424             53           2429
HTML                            13             44             83           1471
-------------------------------------------------------------------------------
SUM:                           158           1171            679          12930
-------------------------------------------------------------------------------

I found it very difficult to get the "big picture": where's the entry point of the entire app? What files call what? How and where should I add my code? How does the build process work? This is much more difficult than implementing some disembodied function somewhere because in order to add the new component I had to get a working understanding of the entire codebase. It took a long time, but it was rewarding to puzzle through it, and---while I won't claim to know the codebase well---I now know enough to implement future features much more quickly.

Another thing I had to learn was how to use version control properly: forking the main codebase, using branches, and making PRs back into main. It also forced me to be more atomic with my commits as I previously had the very bad habit of coding for hours if not days without a single commit, changing tons of files, then running the following:

git add .
git commit -m "add lots of stuff"
git push

which is not good practice, so I'm glad that I kicked the habit.

Future work

Improve performance of the server

The current server is able to quickly calculate and respond in fractions of a second for a state like Iowa (0.007s, according to my server logs), but the delay between sending and receiving a response can be up to 1.5s for the largest states like Texas. This delay is barely acceptable for a single user but completely unacceptable when we deploy it into production e.g. with multiple concurrent users. At the current scale of Districtr, it should be more than sufficient to handle up to 100 requests/second. It could do 100 Iowa requests/second but not 100 Texas requests/second, so there's some work to be done.

I will have to do some benchmarking: why is the server slow? Is it converting the JSON to a dual graph, generating the partition, or sending the response back that's slow? Depending on the answer, we could either consider horizontal scaling (implementing a load balancer) or moving to the very recently developed JuliaChain (GerryChain in Julia). (Maybe also Amazon lambda functions.)

([TODO] do benchmarking and ask Bhushan/Matthew about benchmarking in Julia)

EDIT 28/08: I did the benchmarking and it turns out that loading the JSON file in was indeed what was taking so long:

2020-08-28 10:18:54 /home/mggg/districtr-eda/dual_graphs/mggg-dual-graphs/texas.json
2020-08-28 10:18:54 Time taken to load into gerrychain Graph from json: 0.8254358768463135
2020-08-28 10:18:54 Time taken to form assignment from dual graph: 0.008115530014038086
2020-08-28 10:18:54 Time taken to form partition from assignment: 0.0035495758056640625
2020-08-28 10:18:55 Time taken to get split districts: 0.0813145637512207

Caching the state graph (just adding it to a dictionary and checking if it exists) makes it much, much quicker. Total response times dropped from 0.85 seconds to ~0.1 seconds.

2020-08-28 10:38:51 /home/mggg/districtr-eda/dual_graphs/mggg-dual-graphs/texas.json
2020-08-28 10:38:51 Caching the state graph...
2020-08-28 10:38:51 Time taken to load into gerrychain Graph from json: 0.7195932865142822
2020-08-28 10:38:51 Time taken to form assignment from dual graph: 0.0022351741790771484
2020-08-28 10:38:51 Time taken to form partition from assignment: 0.002564668655395508
2020-08-28 10:38:52 Time taken to get split districts: 0.013953685760498047
2020-08-28 10:38:53 /home/mggg/districtr-eda/dual_graphs/mggg-dual-graphs/texas.json

2020-08-28 10:38:53 Retrieving the state graph from memory..
2020-08-28 10:38:53 Time taken to retrieve the state graph from memory: 4.5299530029296875e-06
2020-08-28 10:38:53 Time taken to form assignment from dual graph: 0.002898693084716797
2020-08-28 10:38:53 Time taken to form partition from assignment: 0.003316164016723633
2020-08-28 10:38:53 Time taken to get split districts: 0.014100790023803711

Conclusion

I really enjoyed my time with MGGG and I learned a lot from implementing a feature onto an existing codebase. I didn't learn much on the technical level, but I did learn a lot about working as an actual software engineer in a team contributing to an established product, rather than as a seat-of-the-pants hackathon lone-wolf developer. This was my explicit goal when applying to MGGG, so I am happy with the outcome.

Appendix

I have collected dual graphs into an open-source repo.

As mentioned in the text, the server cannot generate a Partition without the dual graphs. I've thus collected dual graphs from Gabe and Nick and put them into a repo here. I would welcome any and all to add to the repo to roll this out to more states.

([TODO] --- where is Districtr getting the dual graphs for all the states then??)

How to set up the server on PythonAnywhere

I have written instructions on how to deploy the Flask server on PythonAnywhere on the README.md in the GitHub repo

([TODO] deploy Flask production server)

How the `assignment` object actually looks like

I told a little white lie for ease of exposition. The assignment object actually looks like this:

{
    19001: [2],
    19002: [0],
    19003: [1],
    19004: 2,
    19005: [1],
    ...
}

Each key-value pair takes the form of either an Int: Array[Int] or Int: Int, where the key is the voting precinct ID and the value is the district it has been assigned to. (For some reason, some values will appear as arrays even though each precinct can only be assigned to one district).

A market-based mechanism to preserve Indonesia's forests

2020-08-19T00:00:00Z

Introduction

Enormous swathes of virgin forest in Indonesia are being cut and burned down all the time and this contributes significantly to the climate crisis. How can we stop this? I propose a decentralised market-based mechanism that has the following desirable properties:

Scalable
Decentralised
Bounded cost for the philantropist
Owners of the land are better off

Background

Norway has some sort of agreement with Indonesia where they pay Indonesia $1 billion to reduce their forest emissions: see links here and here.

The two countries signed the $1 billion pact in 2010, under the REDD+ (reducing emissions from deforestation and forest degradation) mechanism. In exchange for the funding, Indonesia would have to slow its emissions from deforestation, which accounts for the bulk of its CO2 emissions.

This is great and should be applauded. But this sort of enormous government-to-government approach has its problems. First of all, what can we do as private citizens, apart from lobbying our governments? Also, this approach is slow: the first payment was paid out almost ten years after the agreement was signed.

That it’s taken so long for the first payment to be announced is due to a combination of the structuring of the agreement and a change in the Indonesian government since the 2010 signing.

I present a market mechanism that might help safeguard the forests, but doesn't require long and protracted intergovernmental negotations and can work with only a few dollars of funding. The idea is probably not original and I would greatly appreciate if someone could point me to the relevant literature.

The mechanism in brief

In brief, the idea is as follows: philantropists can sell a ticket attached to some forested area that pays out if and only if that area remains pristine. People would bid on the price of this ticket and the highest bidder would stand to gain (Ticket Payout Price - Price Paid for Ticket). With satellite imagery, we can easily tell whether an area has been logged or not, and this would serve as a source of truth.

An explanation of the mechanism

Here's how it would work. Suppose we have a philantropist that wants to preserve some area of forest. And suppose his maximum willingness to pay for this area of forest is $50,000. And now suppose we have some satellite feed of this area (e.g. zoom.earth). Then what the philantropist can do is say the following:

"I will pay $50,000 to the person who holds this ticket if and only if $AREA remains forested by $DATE, as determined by satellite imagery. Otherwise, the $50,000 will be used to fund another similar ticket. ^[1]

And then he simply puts this ticket up for auction. Once someone wins the bid, the buyer pays the philantropist. The philantropist then puts his funds in escrow, and some trusted third-party determines whether or not the area has remained forested by the date specified in the ticket---whereupon the funds will either be released to the bearer of the ticket, or used to fund a similar future ticket.

How does this work? The buyer has an incentive to hunt down the farmer/owner of the forest and come to some deal to make sure that the farmer/owner doesn't cut down the land. This will probably take the form of a side payment. And if the farmer is savvy enough he can of course purchase the ticket himself and earn free money from doing nothing at all.

This mechanism has several attractive qualities.

Firstly, it's decentralised: there is no government intervention needed. This is also scalable: any person could conceivably sell such a ticket for as small or as large an area as they want, so if any citizen wanted to be a Keeper of the Grove it they could do it today, without having to lobby their governments to do something. All you'd need is a trusted third party to i) hold funds in escrow and ii) verify the forest cover of the area, both of which could be incentivised with some fees.

FAQs

Why not pay the owners directly? Why the need for this complicated scheme?

How are you going to track down the owners of the land if they live in a forest in the middle of nowhere? Is there a unified land ownership and designation map? Is your Indonesian good enough to find out? How likely is it that that the land ownership map is constantly updated? What happens if ownership has changed hands without your knowledge? What happens if the land is co-owned by several smallholders in some sort of sharecropping arrangement? And so on.

An Indonesian friend of mine wrote to me:

A universal land designation map was initiated in 2014, but it hasn’t come to completion yet because the project has been delayed over and over again.

Rather than having to spend effort to track them down ourselves, the idea is to pay middlemen an "information arbitrage" fee: to provide the service of tracking down the owners of the land. The hope is that these middlemen who buy the ticket will have low-cost ways to contact and convince the farmers.

Why don't we pay NGOs to protect the forest instead?

(Thanks Filip)

Wouldn't this incentivise owners to just sit on the forests and not do anything to collect their payout?

That is the entire point of this mechanism, yes.

Are we worried that if we sell it to the highest bidder, the highest bidder might do unsavoury things?

We might be worried about unintended consequences. For instance, buyers of tickets might start coercing, intimidating, or assassinating farmers to prevent them from cutting down the forest. If so, it might be better to give the money to a good NGO---who will convince/cajole/massage the farmers, rather than to give money to someone unscrupulous---who will murder the owners of the plot---even if the NGO is more expensive.

To this, I have two semi-responses.

The first is that while this might be possible, another possibility is that the NGOs are horrendously inefficient and it might be better to give the money to an amoral corporation instead. Of course, there's no reason why NGOs can't bid on these tickets themselves if they are indeed the most efficient organisation. The other is that killing people is illegal (maybe this will give some people pause). Also, why do you need to kill someone if you can just give them money instead?

Nonetheless, I see the point. Of course when we optimise for price alone there are lots of things that we give up. This is a problem with any sort of market or competitive pricing mechanism, and isn't unique to this proposed mechanism.

What happens if there is collusion amongst the bidders?

The whole system falls apart, but I think any auction system falls apart once there is collusion amongst the buyers. However, if the market is thick enough then collusion would be difficult to sustain. Perhaps there might be some clever way for philantropists to demarcate areas to protect so as to minimise the likelihood of collusion.

Why can't the money go back to the philantropist if the forest burns down?

In a previous version of this mechanism I suggested that the $50,000 would either be released to the buyer, or returned back to the philantropist. I realised that this would incentivise "philantropists" to sell a lot of tickets, then commit arson so as not to pay out to make a profit. Any mechanism must ensure that the philantropist can never gain from this transaction. More strongly, it must ensure that nobody can possibly make any profit if the forest is cut down. (Does this always hold? I am unsure if this holds if people start coming up with complicated derivatives and financial instruments and what not)

A system like this doesn't benefit the poor farmers! Won't the surplus go to some finance bro?

I agree, and while the thought of this is indeed somewhat unsavory, I believe under a competitive market it should be the case that the finance bros will bid each other up until the surplus accrues to the farmers.

Questions I still have

I'd really be interested in knowing whether this mechanism works, and whether it has already been thought of by somebody else under a different name.

Conclusion

Of course, this works with any resource that we want to preserve, not just Indonesia's forests. You can even sell tickets on areas where the land has already been cut down, and pay out iff the land gets reforested. To handle the issues of escrow, we could do it on the blockchain.

Have I missed something? Interested in implementing this? Send me a message, please.

Acknowledgements

Thanks to Filip, Dylan, and Jonathan for reading versions of this article and giving very helpful comments/asking very illuminating questions.

Appendix: Things I cut, links, and text dumps

The key here is to realise that a) the forests are a positive externality in the sense that they sequester carbon and reduce global temperatures, and b) we are much richer than many of the Indonesians who own the forests, and they need to eat.

https://www.newmandala.org/cultivating-consent/

Bernardinus,* a village elder from the Basik-Basik clan, for instance, noted that “oil palm companies come here, obtain government permits and plant oil palm without asking permission from the traditional landowners. They think no one lives on this land because all they see is forest. Sure, Indonesia’s largest remaining forests are in Papua. However, people live in and depend on these forests. Indigenous people like Marind, Mandobo, Auyu, Jair, and many more. We own this land under customary law. And yet our voices are not heard”.

In a similar vein, Perpetua, a mother-of-four living in Khalaoyam village, stated that “Decisions about oil palm projects are made in Jakarta, not in Papuan villages. These decisions are made without our consent or participation. Of course, there are problems and conflict. It’s like planting a tree without making sure the soil is fertile and that there is enough water and shade for the tree to flourish. The tree will grow crooked and struggle to survive. It’s the same with oil palm projects in Merauke. The problem is that there is no consultation or consent”.

Consent on paper, however, does not guarantee consent in practice. Bernardinus and Perpetua are just two of dozens of community members who report not having been involved in or informed about oil palm developments now taking place in and around their home villages. Another is Kosmas, a Marind participant at a workshop on sustainable palm oil that I organized in July 2019, who told me, “I only found out about that oil palm was coming after the forest had already been cleared. Before that, I knew nothing about the project”.

In a small number of cases, government and corporate bodies have invited indigenous landowners to participate in sosialisasi before land developments. But sosialisasi, a term that can be translated loosely as ‘awareness raising’ or ‘public information dissemination’, is also problematic.

Firstly, sosialisasi often takes the form of a one-way transfer of information about the planned development from the government and companies to the communities, rather than a two-way conversation in which communities have a say in whether or not the project will go ahead, and under what conditions. Secondly, sosialisasi usually takes place in urban centres, such as Merauke City or Jayapura, rather than in the villages where indigenous communities are based. Those invited to attend these events are often handpicked by the government and companies and do not represent the broader interests or rights of the communities to which they belong. Thirdly, sosialisasi meetings are frequently supervised or attended by members of the police and military, who work with oil palm companies in the capacity of plantation security patrols and brokers. Many Marind are reluctant to speak out freely under such conditions for fear of consequent reprisal.

Meanwhile, Marind villagers who have already voiced their opposition to oil palm projects during consultations complained of being persistently approached, and sometimes intimidated, by company personnel and police officials to surrender their lands for agribusiness development. “They keep coming and asking us to change our minds,” reported village elder Geronimo. “We already said no, but they think that if they keep pushing us, we will agree to collaborate. That is not seeking consent. That is harassment”.

Why this harassment? Incentives not aligned. At the moment, oil companies want to skirt the law any way they can. If we can pay the oil palm companies or local governments more money to leave the land intact, this problem solves itself.

The $50,000 needs to be "burned" (to use a crypto term) somehow. It cannot go back to the philantropist/to charity or to any third party, because then that person will have a perverse incentive to make sure the area of land is cut down in order to receive the payout. ↩︎

Rebuilding this website

2020-08-03T00:00:00Z

Introduction

Restoring existing functionality

Learning the templating language

Writing custom filters

Parsing Markdown snippets

Maintaining backwards compatibility with old permalinks

"Cool URIs don't change" consistently emphasises the need for URIs not to change, mainly because someone may have bookmarked your post's old URL and it would be terrible if they couldn't find it any more. Eleventy's default paths when generating posts was different from Jekyll's, so I made sure to point the URLs to the same place by putting

permalink: "/{{ page.date | date: '%Y/%m/%d' }}/{{page.fileSlug}}/"

in the YAML front matter.

The "Cool URIs don't change" post recommends indexing by the creation date of the document, as it is one thing that doesn't change. "If a document is in any way dated, even though it will be of interest for generations, then the date is a good starter." It recommends a URI like http://www.w3.org/1998/12/01/chairs, which is indeed what Jekyll uses by default.

The post further writes "the only exception is a page which is deliberately a "latest" page for, for example, the whole organization or a large part of it", for instance, http://www.pathfinder.com/money/moneydaily/latest gives the latest version of the "Money daily" column in "Money" magazine. If you want to link to the content, however, one would link to it where it appears separately in the archives as

http://www.pathfinder.com/money/moneydaily/1998/981212.moneyonline.html.

In my case the "latest" pages are the index pages: home, about, projects, archive, etc, and most of the rest of the blog posts are dated.

For some of my work, I am considering following Gwern's example to make URIs indexed by a topic sentence: for instance,

https://www.gwern.net/Ethical-sperm-donation

links to a page entitled "The Morality of Sperm Donation". The pages are constantly updated: while the page was first written in 2012, the last major update to the page was in 2018. This makes sense for Gwern's "long content": the date of document creation is irrelevant because the pages are constantly updated. I don't have much long content on the site yet, but I hope to start writing some soon. I think the explorations and explanations would be good candidates.

Improving the website

Adding a dark mode

Revamping the projects page

Supporting Markdown table of contents and footnotes

Rendering LaTeX, and moving from MathJax to KaTeX

Things I still need to do

Fix broken post links

Fix broken image links

Sites and blogs I drew reference from

Practical Typography

gwern.net

Reasonable Deviations

Iain Bean

Conclusion

Building a code sharing MVP with React and the WebSockets API

2020-08-02T00:00:00Z

Introduction

Task

Introduction

ThirdSpace is a real-time classroom environment where students can:

Execute and submit code to instructors.
Access files and resources (homework) from instructors.
Chat with each other, and ask questions.
View real-time or pre-recorded lectures.

Instructors have separate views where they can:

View submitted code from every student.
Share their code to the students code editor.
Post files and resources (homework).
Create courses with scheduled classes.

Questions

To the best of your abilities, please design a scalable system architecture for ThirdSpace. No code needed.
Build a simple application with ReactJS and ExpressJS. You may use any JS libraries to your liking. Please see the instructions here.

My attempt

Question 1 : Designing a scalable system architecture

I break the monolithic environment into separate services:

Code share+submit service: Instructors enter private rooms with students, and can share solutions with students. Students execute and submit code.
Code viewing service: A static view for instructors to view submitted code
Chat service: Real-time chat.
Course view service: largely static service to display text, images, and videos.
Live stream service: dedicated service to encode streaming media and chunk it

I'll now go through the overall architecture and show where the individual services fit in.

The overall architecture uses Amazon's API Gateway, AWS Lambda, and Amazon DynamoDB for scalability. We use these services as we get scalability "for free", although I also sketch out non-AWS alternatives.

Upon authenticating, students enter the ThirdSpace classroom environment. The student's client establishes several WebSocket connections to an Amazon API Gateway. The following WebSocket connections are established through the API Gateway:

Chat service WS
Code sharing service WS

Additionally, the client also makes the appropriate GET request to the Course view service, which will display the courses that the students are involved in.

Suppose the student now wants to chat. He will then send messages through the established WebSocket to the API Gateway which calls an AWS Lambda function. The waiting AWS Lambda function calls Amazon DynamoDB for the WebSockets subscribed to the chat channel, and callback to the API Gateway. The API Gateway then sends payloads through all the other WebSocket connections.

The same principle applies to the code sharing service (we build an MVP in section 2), which is very similar to the real time chat service. Once again, students send code through the WebSocket which calls an AWS Lambda function. The difference is that this function may need to compile the submitted code (only Javascript can be interpreted client-side) as well. As before, it calls DynamoDB, and returns payloads to the teacher's Websocket connection.

The course view service can be handled with a normal REST API. We can either use Amazon's API Gateway with Amazon DynamoDB or a more traditional architecture running on a server (or multiple servers).

If we want a more traditional approach: students send GET requests to the server, and instructors who wish to upload course materials can make a POST request to the server. We can ensure scalability by using a load balancer, and course materials can be stored in some shared database.

Finally, live streaming should be handled with HTTP Live Streaming (HLS). The following diagram from Apple's developer documentation lays out all the moving parts:

We should use a dedicated server to receive AV input from instructors, do the encoding and stream segmentation (this could be farmed out to Lambda functions) and serve them out to the clients.

The code for the app is on a private repository here.

This is what an old version of the app looks like (GIF):

There is a React app running on port 3000 and an Express server running on port 3333. Clients communicate with the server using a bidirectional WebSockets connection. The server contains a list of Rooms and can handle actions (instead of routes) such as join (a room), leave (a room), run (some code), and share (some code). We can see that the teacher's view updates in real-time as the student submits his or her code.

Below is the backend schema:

Room

This lives in memory but should be stored in a datastore.

class Room {
  id: string;
  teacher: Username;
  student: Username;
  sharedCode: string;
  teacher_ws: WebSocket;
  student_ws: WebSocket;
}

Actions

"JOIN" -- join a room
"LEAVE" -- leave a room
"RUN" -- run some code (Student view only)
"SHARE" -- submit some code to the Teacher in the room (Student view only)

Schema for each WebSocket request from client --> server

{
  username: Username (just a string),
  action: "JOIN" | "LEAVE" | "RUN" | "SHARE",
  roomId: RoomID (just a string) | null
  role: "Student" | "Teacher" | null

  // This is action specific
  // If action is "JOIN":
  payload: {
    roomId: RoomID
    role: "Student" | "Teacher"
  }

  // or if action type is Leave, payload is
  payload: {
    roomId: RoomID
    role: "Student" | "Teacher"
  }

  // or, if action type is Run, payload is
  payload: {
    code: a JS string that can be run
    }

  // or, if action type is Submit, payload is
  payload: {
    shared_code: a JS string corresponding to input code
    }
}

How does the server respond to each type of action?

If action === "JOIN":

Look at the payload and check whether:

The room exists
If the room exists, whether the role is taken

Return the Room object or an error message

If action === "LEAVE" (NOT IMPLEMENTED):

check whether the person is actually in the room with that role
return either some acknowledgement or return an error message

If action === "RUN" (IMPLEMENTED BUT NOT USED):

Take the payload code and run it on an interpreter
Return the outcome of the code (could be an error, or some output)

If action type === "SHARE":

Take the payload code and broadcast it to the other person in the room

What sort of messages can a client expect to receive?

{
  responseType: "JOIN" | "LEAVE" | "RUN" | "SHARE" | "ERROR" ,
  room: Room,

// again, action dependent
// if response_type === "JOIN"
// empty
  payload: {
    role: "Student" | "Teacher"
  },

  // if response_type === "RUN"
  payload: {
    codeOutput: string,
  },

  // if response_type === "SHARE"
  payload: {
    shared_code: string,
  },

  // if response_type === "ERROR"
  payload: {
    error_message: string,
  }
}

FAQs

Why no tests? I didn't put them in due to time contraints, but I should have.

Why no routes (GET/POST)? Given that the app needed real-time bidirectional communication between client and server, I decided to use WebSockets. And if we use WebSockets we don't really need routes because WebSockets allows us to send arbitrary JSON very quickly.

Singapore's GRC system likely advantages the PAP: a game-theoretic analysis

2020-07-07T00:00:00Z

(with huge thanks to Tsun Lok, Bassel and Chin Wee, and thanks to all who proofread early drafts and gave helpful comments).

~2000 words

Overview

I develop a game-theoretic model of electoral contestation in Singapore and show that having multi-member GRCs rather than only single-member constituencies (SMCs) gives the incumbent party more seats if the incumbent's "strong" candidates are significantly stronger than the opposition's ones.

Given that this assumption is likely to be true, this suggests that the GRC system advantages the PAP and gives it more seats than it would otherwise win under a "pure" single-member constituency system like the UK's.

Background

First---what's a GRC? From the Wikipedia page:

A Group Representation Constituency (GRC) is a type of electoral constituency (unique to Singaporean politics. They are multi-member constituencies (usually 4, 5 or 6 seats) which are contested by teams of candidates from one party or from independents.

In each GRC, at least one candidate in the slate must be from a minority race (ergo non-Chinese). The ostensible purpose of the GRC system is to ensure minority representation. It may well do that, but might it inadvertently benefit the PAP as well?

GRCs operate with a plurality voting system voting by party slate, meaning that the party with the largest share of votes wins all the seats in the GRCs. But this may make us ponder. A strong candidate can single-handedly win all the seats in the GRC. For instance, if the PAP fielded four chimpanzees and the late MM Lee in a five-member GRC, it would handily win that GRC (and thus five seats) through the force of MM Lee alone. In contrast, an electoral system that only contained single-member constituencies (SMCs) would mean that MM Lee could only ever win one seat by himself no matter how great he was.

The reasoning I've put forth is quite intuitive: strong MPs can "carry" weak ones in the GRC system but not the SMC system. But how do we formalise this intuition? And how large is this effect, if any? I come up with a toy game-theoretic model to answer these questions. I show not only that the GRC system gives the PAP an advantage, but also the extent of this advantage---and that that this advantage works through magnifying existing advantages that the PAP already has.

Building a game-theoretic model

First a preliminary note. When modeling it is vitally important to make the model as simple as possible---and then even simpler than that. Otherwise the problem becomes completely intractable. And thus this model leaves out many important considerations by necessity.

With that said: In any election there are stronger candidates and weaker candidates. For simplicity, let us consider two types of candidates: (1) eloquent, long-serving, well-respected candidates (minister-calibre candidates) and (2) other candidates (perhaps-currently-not-minister-calibre candidates). For example in Jurong GRC, one might think that Tharman and Xie Yao Quan would respectively exemplify the qualities of each category.

Let's now assume that a "weak" candidate has a strength of 0 and a "strong" candidate a strength of 1. Let's further assume the PAP's stronger candidates are stronger than the Opposition's candidates. This can be thought of as an "incumbency bonus" and you can interpret it in any way you find reasonable. I am partial to three main explanations for this incumbency bonus:

Firstly, the PAP's political dominance may give it a recruiting advantage (as minister-calibre candidates may not want to risk their careers on joining Opposition).

Secondly, the PAP's political longevity gives it an incumbency advantage. If two candidates are identical but one has been a minister for ten years and the other an Opposition MP for five, the former will be perceived as stronger by the electorate.

Lastly, the relative friendliness of state media to the PAP's interests can also give its candidates an incumbency advantage. Whichever explanation you prefer, let's say that "strong" PAP candidates have strength 2 but "strong" Opposition candidates only 1.

Now consider this game.

The 2020 General Elections has 14 SMCs, 6 four-member GRCs, and 11 five-member GRCs, for a total of 93 (14 + 6 * 4 + 11 * 5) seats. This is a total of 31 districts. Each party wants to find the optimal assignment of strong and weak candidates over all 31 districts such that:

in each district, the party that has allocated the higher sum of candidates' strength will win all the seats in that district;
both parties seek to maximise the total number of seats they expect to win.

(those with a background in game theory may recognise this as a variant of the Blotto game).

This can be a bit confusing, so let's run through a tiny example. Let's suppose that there are only two districts: Yuhua SMC and Jurong GRC. Let's further suppose that the PAP has two strong candidates (Grace Fu & Tharman) while the opposition only has one. If the PAP chooses to field one strong candidate in each district, its strength in each district is:

Yuhua SMC: 2
Jurong GRC: 2

But if it chooses to field both strong candidates in Jurong GRC, its strength in each district is:

Yuhua SMC: 0
Jurong GRC: 4

Now suppose the opposition chooses to field its strong candidate in Yuhua SMC. Its strength in each district is therefore:

Yuhua SMC: 1
Jurong GRC: 0

Then if PAP had chosen to field one strong candidate in each district it wins all six seats, because it wins Yuhua SMC (2 > 1). But if it chose to field both strong candidates in Jurong GRC it would win only 5 seats as Yuhua SMC would be lost to the Opposition.

Hopefully that clarifies.

One final point: this game can be played either simultaneously or sequentially. In the simultaneous version of the game, both parties choose where to send their MPs at the same time with no idea what the other party is doing. In the sequential version of the game, the PAP first chooses how to field its candidates. Then the Opposition observes the PAP's move and then chooses how to field its candidates. The rationale for this is that the opposition has some idea of which incumbent PAP MPs are in which area, and whether they are strong or not.

In this exploration I use the sequential game as it is (1) easier to solve and (2) I suspect that the results will be even more skewed in favour of the PAP if the Opposition has no last-mover advantage (see Technical Appendix). That is to say, if we find a PAP advantage when the Opp gets to observe the PAP's actions, we should strongly suspect that this advantage will increase when the Opp doesn't get to observe the PAP's actions.

So that's the game-theoretic model. To reiterate:

Each party is endowed with a different number of strong and weak candidates. The PAP's strong candidates are worth 2 while the Opposition's candidates are worth 1.
First, the PAP assigns strong and weak candidates to all 31 districts.
After having observed the PAP's assignment, the Opposition assigns their strong and weak candidates to those 31 districts.
We compare the strength of each party in each district and add up the expected number of seats won by every party (Draws give half the seats of the district to both parties).

We've just specified the game, and can now analyse it.

Methodology

The full electoral situation has 31 districts and 93 seats---too large to analyse quickly---so I decided to shrink it by one-third to make it tractable. In my reduced game, there are six SMCs and five 5-man GRCs, for a total of 31 (6 * 1 + 5 * 5) seats. I made sure to keep the ratio of SMC and GRC seats roughly similar to the full electoral case.

We need one more piece of information to solve the game: the number of strong and weak candidates that the PAP and Opposition have. You could attempt to use your judgment to identify this number, but this would introduce human bias; instead, I try a variety of different combinations to make sure my results are robust.

I didn't try to do anything fancy when solving the game: I just used a brute force approach. A sequential game with perfect information can be solved with backward induction. Here's how I went about it:

I first generated exhaustively all possible ways the PAP could assign candidates to districts. I then did the same for the Opposition.

Then, for each of these PAP-assignments, I looked at each of the Opposition's assignments and selected the Opposition's optimal strategy: that is, I found the Opp-assignment that would give the most seats given that PAP assignment.

This told me what the Opposition's best response/optimal strategy would be given each PAP assignment.

Finally, I looked at all the PAP-assignments and Opp-best-responses, and chose the assignment that maximised the PAP's number of seats.

All this was done with Python in a Jupyter Notebook and can be found here (warning: very rough).

Results

Including GRCs gives the PAP more seats compared to a pure-SMC electoral landscape. Figure 1 plots the number of seats the PAP wins under both types of electoral systems with different combinations of strong candidates.

The first table shows the number of seats won under the pure-SMC electoral landscape (i.e. 31 SMCs). For instance, the top left hand corner shows how many seats the PAP would win if it had five strong candidates and the opposition had two. The PAP puts its five strong candidates anywhere and the Opp judiciously avoids those strong candidates. The end result is that the PAP wins five districts, loses two districts, and draws the remaining (31-5-2 = 24) districts, winning a total of (24/2 + 5 = 17) seats out of 31.

The second table shows the number of seats won under a landscape with 6 SMCs and 5 GRCs. Again, let's look at the top left hand corner as an example. The PAP's optimal strategy is to put one strong candidate in each GRC (thus each GRC has strength 2). The Opposition chooses to put both strong candidates in one of the GRCs to contest it, causing a draw. The PAP thus wins half of the six SMCs, four GRCs, and half of the final contested GRC, for a total of (6/2 + 4 * 5 + 5/2 = 25.5) seats.

While in this case the PAP has more strong MPs than the Opposition, we can see that the PAP benefits under the GRC system regardless of the number of strong candidates. In fact, the PAP wins more seats (17.5 vs 14.5) even when it has a numerical advantage (5 strong PAP vs 7 strong Opp). The advantage is magnified when there is a large discrepancy between the number of strong candidates each party has.

Caveats

Firstly, my results depend strongly on the incumbency strength bonus and the relative ratio of strong MPs for each party. Here each strong PAP candidate is worth two opposition ones. With a low enough incumbency strength bonus, the last-mover advantage the Opposition has can give them the advantage.

Figure 2 shows the results of subsequent analyses I ran with a lower incumbency strength bonus (1.5 vs 2) and a larger range of strong PAP candidates. We can see that the GRC may actually disadvantage the PAP if the number of strong PAP candidates is significantly lower than the number of strong Opposition candidates. It's more accurate to think of the GRC system as a "variance-increasing" system.

However, I don't think looking at the lower end is that interesting, because (1) it seems unlikely that the PAP has significantly fewer strong candidates than the opposition, and (2) I strongly suspect that the PAP disadvantage will disappear completely in the simultaneous game without the WP's last-mover advantage.

Secondly, the results may be specific to a particular range of strong/weak MPs or a particular makeup of SMCs and GRCs. I've tried to ward this criticism off by trying multiple combinations but may have missed something. Nonetheless, I'm quite sure---but not 100% certain---that this analysis will hold for the larger 93-seat game.

Finally---to be clear, there is nothing inherent about the GRC system that benefits the PAP. It is only when paired with the assumption of incumbency advantage (and a reasonable number of strong v. weak candidates) that the GRC system becomes structurally favourable for the PAP. This insight is important, because it allows us to infer whether or not the PAP intentionally benefits from the GRC system. If it is, then we should expect to observe either a decrease in GRC size or an increase in the number of SMCs relative to GRCs in the future if the Opposition gains in relative size and strength.

Conclusion and discussion

Provided that the incumbency strength bonus is high enough, the GRC system advantages the PAP by letting it win more seats than it would have under a traditional single-member system. We should thus carefully consider whether the purported benefits to minority representation outweigh the increased disproportionality that results.

Future research should try to solve the simultaneous version of this Blotto game, and try to find a relationship between the incumbency strength bonus and the number of extra seats won.

Technical appendix

Claim: The simultaneous Blotto game will always give the PAP more seats compared to the sequential Blotto game.

Proof: Is this actually true? TBD. (Ask Bassel for help).

My 2020 fitness routine

2020-07-01T00:00:00Z

Introduction

This is going to be a reference page so I know what to do and why I'm doing it.

The goal is for my routine to be zero-friction: make sure I exercise every day and maintain the meta-stable habit. Otherwise I will definitely slack off.

Goals

My priorities are to shore up my weaknesses and then build upon my strengths. In decreasing order of priority, therefore, my goals are:

Fix my anterior pelvic tilt
Fix my internal shoulder rotation
Increase flexibility
Increase strength and hypertrophy

To that end, I'll be doing the following:

Stretch every day with Starting Stretching (SS) routine
(For internal shoulder rotation) Add doorway face pulls, subscapularis and pec minor stretches to the SS routine
(For APT) Add modified sprinter lunge to the SS routine
Perform 6-day push-pull bodyweight routine

Anterior pelvic tilt

I have mild anterior pelvic tilt (APT). The natural angle of anterior pelvic tilt should be 3-4 degrees downward on the waistline, mine is a bit more than that, I think.

To remedy this, I'll be following Athlean-X's tips for anterior pelvic tilt (video here].

Don’t stretch hamstrings
Strengthen glutes with the "modified sprinter lunge" shown in the video (around 4:31)
Stretch lower back
Stretch hip flexors (part of Starting Stretching, more later)
Strengthen abs without engaging hip flexors

Strengthening the glutes and stretching the hip flexors

Strengthening the abs

I will be doing two main exercises that don't strengthen the hip flexors because training the hip flexors can exacerbate APT. Athlean-X link:

Many different ab exercises enlist the help of the hip flexors as a prime mover. This occurs a lot in lower ab exercises but can also occur in things as simple as the crunch. If you allow your ab training to include too many of these you will wind up simply over strengthening the hip flexors more and leads to a vicious cycle of more pain.

I've chosen two main exercises from the video to do:

Side plank (with optional leg lift)
Gymnast "captain's chair" back protraction

I will be doing these ab exercises in my daily hypertrophy routine (more on this later).

Internal shoulder rotation

I have very bad internal shoulder rotation due to a combination of i) benching too much without doing as much horizontal pull work and ii) spending too much time in internal shoulder rotation (typing on keyboard)

Again following Athlean-X's recommendations

Editing a YouTube comment here:

2:50, Thoracic spine stretch: Stomach on ground, arms above head 45* angle, rotate back while keeping chest wide open, working your way to the left and right side
5:48, Subscapularis stretch: 30 seconds: externally rotate arm as much as you can, hook arm against doorway. While arm is kept back way rotate your chest out in the other direction
07:08, Pec minor stretch: hold shoulder back against door, pinch shoulder blades together, raise arm up
08:26, Band pull-aparts: Externally rotate arms away with thumbs pulled back, keeping shoulders/traps down, 20 sets of 1
10:27, Band serratus/lower trap exercise: Whilst holding a resistance band, protract your arms (without rounding the back). Externally rotate arms out just enough to feel band tension. Raise above the head, down, reset, repeat

I'm going to do the first three stretches every day as riders to the SS routine.

I will also be doing doorway face pulls. Face pulls strengthen the posterior deltoids which should help pull back the shoulders.

Flexibility

I am very inflexible. I can't touch my toes, my hamstrings are very tight, my shoulders are tight, everything that can be tight is tight. I would like to do the splits eventually. Here's what that looks like now:

I'm doing Starting Stretching every day, but skipping the hamstring stretch. Here's an infographic:

Strength and hypertrophy

I'll be starting with a Push-Pull bodyweight bodybuilding routine that I found on /r/naturalbodybuilding, exercising six times a week.

The PDF is in this directory and the Reddit post is here.

Pull

Heavy vertical pull 2-5
Medium intensity horizontal pull 2-5
Bicep isolation 2-5
Side delt isolation 2-8
Heavy hamstring exercise 2-5
Medium intensity hamstring exercise 1-5
Abs 3

Push

Heavy horizontal press 2-5
Medium intensity incline press 2-5
Triceps isolation 2-5
Heavy quad compound 2-5
Medium intensity quad compound 2-5
Straight legged calf exercise 3-8

For ab work I'll be doing the exercises I wrote in the previous section: these are

Side plank (with optional leg lift), and
Gymnast "captain's chair" back protraction.

I'll be replacing some of the exercises with dumbbell work: for instance, I'll be using DBs for lat raises, and could consider DB squats if pistol squats get too easy.

I might also consider working in vertical presses sometimes to replace the incline press e.g. DB shoulder press.

Warm-up

Pasting directly from the Reddit post for reference:

Do this before every session:

Part 1: Elevating your core temperature.

Here you can either do 3-5 minutes of LIGHT cardio (brisk walk/bike ride/elliptical), or super light full body exercises like the ones you can see in this video. Since the only objective here is elevating your body temperature, you absolutely don't want to get tired, out of breath or sore, just stop when you feel like you're warm enough. Breaking a sweat is ok.

Part 2: Dynamic warm-up.

Here you want to move explosively through the ranges of motion you're going to be using in the workout. Follow this protocol:

10 leg swings front to back (explosive).
10 leg swings side to side (explosive).
10 big arm circles forward (explosive).
10 big arm circles backwards (explosive).
10 cross body arm slaps (explosive).
5 repetitions (per leg) of a lunge with torso rotation (only if for you it's a light exercise and doesn't tire you).

After that, the regular training session starts. Then, do the following before every exercise that's going to work a muscle group for the first time in the session (for example if in the same session you have two exercises for the chest, then do the following only before the first one):

Part 3: Sport-Specific Warm Up.

Do a set of 5 reps from the previous level of the exercise you're about to do. So if you're warming up to do the horizontal press level 3, in this instance do one set of 5 reps of the horizontal press level 2. If you're warming up for a level 1 exercise, and its intensity can be lowered as explained here, do the lower intensity version; and if the intensity can't be lowered, do 5 partial reps: 2 partials from the bottom part of the movement, and 3 partials from the top part of the movement. If you're about to do an exercise without a specific level, use a lighter exercise that targets the same muscle group.

Rest for a few seconds and then do your working exercise for half of the reps. So if you're warming up to do incline push-ups, and last time you got 12 reps in your first set, then in this instance do one set of incline push-ups for 6 reps. Rest for a minute, and start your regular working sets.

Conclusion

My current routine is a big departure from my earlier days where the only thing I wanted to do was get big with as little effort as possible. I suppose I'm paying the price now with the imbalances that have occurred as a result.

How many solar panels would be needed to 100% solar-power Singapore?

2020-06-02T00:00:00Z

Principal investigators: Michael and Celine

Summary

We use horizontal irradiance data, Singapore's half-hourly power demand statistics and solar panel schematics to calculate the area of solar panels (and battery storage) needed to meet Singapore's energy demands. We calculate that an area of 30,000 hectares would be more than sufficient, provided we have enough battery storage.

I'm certain that I've missed something: all feedback and corrections are most welcome.

Thanks very much to Michael and Celine for researching this problem with me: whatever errors remain are mine alone.

Background and motivation

I saw this infographic from the Straits Times talking about an ambitious plan to build a huge solar farm and cable the energy over to Singapore. This piqued my interest, so I went to the company's website to learn more.

What I found puzzled me. The Straits Times infographic claims that the farm will occupy 15,000 hectares and provide a third of Singapore's energy. But the website claims that the farm will occupy 12,000 hectares and provide a fifth of Singapore's energy.

So which is it? 1/5th or 1/3rd? No matter how you slice it the numbers don't make sense. So I decided to take matters into my own hands and calculate it myself. I want to find out: what area of solar panels would we need to provide 100% of Singapore's energy?

Strategy

We'll tackle this question in two main parts: on the supply and on the demand side.

How much energy does Singapore demand?
How much energy can be produced per unit area?
- How much energy falls onto the ground per unit area?
- How much of that energy can we soak up per unit area?

Demand-side: how much energy does Singapore demand?

How much power does Singapore demand? The answer depends of course on "over what period"? Because solar power varies greatly throughout the day, we should take the highest-resolution snapshot of the data.

The best I could get was half-hourly power demand figures obtained from Singapore's Data.gov.sg. The dataset starts in 2012 and ends in 2017 but I extrapolated it to 2020 (by taking the geometric mean).

The chart below plots how power demand varies throughout the day in Watts. Pick any date from 22 Feb 2020 to 20 June 2020 to see how power demand varied during that day.

Surprisingly, power usage tends to dip during working hours and go up during the wee hours of the night. I'm not sure why this is the case: maybe I made a mistake?

Supply-side

How much power can we collect from the sun? If we know the amount of sunlight that shines on a unit area, and we know the efficiency of the solar panels, then the total power collected is

TotalPower = Power/m^2 * Area * Efficiency

And as long as the total power produced exceeds the power demanded by Singapore, we're in the green!

It's not that easy, of course. The amount of sunlight incident upon a unit area depends on things like weather conditions and the time of day. And efficiency depends on ambient temperature and the angle of solar panels, among other things. We'll have to make several simplifying assumptions to proceed.

How much energy hits the ground? Global Horizontal Irradiance (GHI)

From Wikipedia:

Solar irradiance is the power per unit area (watt per square metre, W/m2), received from the Sun in the form of electromagnetic radiation.

Irradiance on the Earth's surface depends on the tilt of the measuring surface, which we'll account for later. At the moment, we'll use a simplifying measure: that of horizontal irradiance (GHI), which measures the power received by a horizontal surface per unit area.

We need to get time-series GHI data because solar irradiance changes dramatically throughout the day (the sun rises and sets, after all). I got the GHI data from the DKA Solar Centre. The DKA Solar Center describes the data in the following way:

A series of four Class A meteorological stations across the Northern Territory collect high resolution solar and climate resource data from locations in Darwin, Katherine, Tennant Creek and Alice Springs.

I used the data from the Darwin meteorological station, but on reflection I should've used Tennant Creek because that's where the site would have been built.

How much energy can we collect?

Knowing how much energy hits the surface is one thing, but what we really care about is how much energy we can collect. This is the most finicky bit of the calculation, because there are many many different factors that affect efficiency. To make the problem tractable we group the losses into three, in decreasing order of importance:

The "base" efficiency of the solar panel
The angle of the solar panels (in appendix)
All other losses (e.g. dirt/soiling/conversion/temperature/whatever)

Feel free to skip any of these sections as they are quite finicky and not that important for the big picture.

Efficiency of the solar panel

I calculated the "base" efficiency of the solar panels to be 16.4%. That is, for every 1kWh of energy that hits the ground, the solar panel would produce 0.16 kWh. I got this number as follows:

Referencing the spec sheets given by Sun Cable's solar panel supplier, each solar panel module has a width of 5550mm and a length of 37330mm. The area is therefore 207.8 sq m.

How much power does it generate? According to the spec sheet, the module produces 34.2 kWp. But what sort of unit is this? It turns out that kWp stands for "kilowatt-peaks", or nominal power, and we'll need this value to calculate the efficiency.

From Wikipedia:

The nominal power of PV devices is measured under Standard Test Conditions... Specifically, the light intensity is 1000 W/m2, with a spectrum similar to sunlight hitting the earth's surface at latitude 35°N in the summer (airmass 1.5), the temperature of the cells being 25 °C.

And so we can get the efficiency of the solar module by dividing the kWp by the area of the solar module. This module produces 34.2 kWp at peak with 207.8 sqm which means that the efficiency is 16.4%. That is, if irradiance were $1kW/m^2$ , you'd get $0.16kW/m^2$ from the solar panel.

What does this mean? It means that for every m^2 of solar panels we install, we would get 0.16kW of power per 1kW of horizontal irradiance. This is exactly what we need: just multiply the horizontal irradiance data by 0.16. Of course, there are still other power losses, which we'll now consider.

Miscellaneous losses

There are a ton of other miscellaneous power losses. The SolarGIS website lists "solar radiation, air temperature, and terrain" as factors in calculating energy output, as well as "losses due to dirt and soiling" and the "cumulative effect of other conversion losses".

In the simulation, losses due to dirt and soiling was estimated to be 3.5%. The cumulative effect of other conversion losses (inter-row shading, mismatch, inverters, cables, transformer, etc.) is assumed to be 7.5%. The power plant availability is considered to be 100%.

The SolarGIS website gives $3.5 + 7.5 = 11%$ , but here we'll be conservative. We'll take an additional fudge factor and say that 15% of energy is lost. And thus the ratio of power produced (PVOUT) to incident power (GHI) is

PVOUT = GHI * 0.164 * 0.85 .

We'll use this value henceforth.

Supply-side graph

In the previous section we calculated that $13.9\%$ (0.164 * 0.85) of energy from the sun would be converted into usable energy. The following graph displays how GHI and PVOUT vary throughout the day per unit area.

Again, you can use the date scrubber to view all the days in the dataset. Check out 9 March 2020: it seems to have been a particularly overcast day,

Putting it all together

We've built the demand and supply sides, and all we need to do now is to put them together! The graph below superposes the demand and supply graphs.

The supply graph starts off tiny, but grows as we increase the area of solar panels. Move the slider to increase the area of solar panels and watch the supply graph grow. Assuming we have enough battery storage, we have enough solar panels to fulfill Singapore's peak power demands when the area under the red curve (total solar energy that is collected for a given area) exceeds the area under the blue curve (total power demand of Singapore).

I've calculated the difference in these two areas (total net supply) in MW. Using the slider, increase the area of solar panels until the total net supply becomes positive.

Depending on the day's weather conditions, the area of solar panels needed to generate more power than demand also varies. Try using the date picker again.

The graph belows shows the minimum area of solar panels needed to meet daily demand. Increase the area of solar panels (red line) to see how much area we would need to meet demand on most days.

We can see that at 40,000 square hectares we would have a huge surplus on most days and would only fail to meet demand for three days in the entire period. With battery storage one could bank up the surplus energy and draw it down during cloudy days.

How does that area actually look like?

In the image below, I've marked in red the square area of solar panels needed to fully power Singapore (~30,000 hectares). This is what that looks like (map from Google Maps):

A length of 100 pixels corresponds to a 500km distance. 1 pixel therefore represents 5km, and a 1x1 pixel square represents an area of 25 sq km. And 30,000 hectares is 300 sq km, which is a 3x4 rectangle---which is what I've drawn on the map.

Zooming out to the entirety of Australia:

Australia is huge: all the solar panels would occupy an area only one-tenth the size of Darwin (3600 sq km).

The area of Singapore is (very generously) 900 sq km, so this corresponds to about a 6x6 pixel square. Placing the solar panels in Singapore, they'd occupy more than one-third of our land area (so not a great idea). Here I've zoomed out and marked the (approximate) square area of Singapore:

Conclusion

Building a solar farm to power all of Singapore is actually not that ambitious. The largest extant solar farm as of 2020 is the Pagavada Solar Park in India (2050 MW), which covers an area of 53 sq km. Our hypothetical solar farm is six times that---which sounds big---but India's building these solar parks like nobody's business. The Bhadla Solar Park (2245 MW) is 45 sq km, and the upcoming Dholera Solar Park is a whopping 5000MW (which will be ~ 100 sq km). The solar park proposed by Sun Cable is only a bit larger than Dholera Solar Park, and the solar park that would meet Singapore's energy demands is only two or three times the size: well within the realms of possibility.

Future work should calculate the amount of battery storage needed to store enough surplus so as to handle bouts of inclement weather.

Appendix

Things that I didn't deem important enough to put into the main text

Tilt of the solar panels

The tilt of the solar panel affects power generation.^[1]

It turns out that "for a fixed tilt angle, the maximum power over the course of a year is obtained when the tilt angle is equal to the latitude of the location" (source: PVeducation.org)

The spec sheet says that the angle of these solar panels is 20 degrees and the latitude of Tennant Creek is 19 degrees, so we're basically as close to optimal as we can get.

Transmission losses

Wikipedia Super Grid Article says that the latest generation of High-voltage direct current power lines can transmit energy with losses of only 1.6% per 1000km:

Sun Cable is using a High Voltage Direct Current Lines, which have quoted transmission losses as less than 3% per 1,000km. (Source: Wikipedia)

The distance between Singapore and Darwin, Australia is around 4,000km.

Thus, we’d get around $4.0*3\% = 12.0\%$ losses - one-eighth of the energy produced would be lost on transport.

Storage losses, batteries needed, and how much area those batteries would occupy

TODO: not that much, batteries are pretty dense. Wouldn't add more than 10% of the area of the solar panels.

From random Googling, battery storage seems to be 85% efficient.

source ↩︎

Theopol Final Exams

2020-06-01T00:00:00Z

You can find the PDF here

QE Final Exams

2020-06-01T00:00:00Z

You can find the PDF here

PolSoc Final Exams

2020-06-01T00:00:00Z

You can find the PDF here

Micro Final Exams

2020-06-01T00:00:00Z

You can find the PDF here

Macro Final Exams

2020-06-01T00:00:00Z

You can find the PDF here

Game Theory Final Exams

2020-06-01T00:00:00Z

You can find the PDF here

Politics Thesis

2020-04-23T00:00:00Z

tl;dr summary here Thesis on GitHub here

Gibbs Prize for best thesis.

Very proud of this work. I spent almost an entire year, and hundreds of hours on this. It couldn't have been done without the time and effort Bassel, Filip, Martin, Tak Huen, Sergi, Andy, Jonathan Rodden, Nick Eubank, and Prof. Stephanopoulos lavished on me.

Comments from markers here and here.

Comments from Stephanopoulos (via email):

Hi Zhenghong,

Just a quick note that I read, and was very impressed by, your thesis. I think it's a remarkable piece of work: one of the first pieces to explore the tradeoffs between different redistricting goals using simulations. If I can assist in any way down the road, please let me know.

On Thu, May 7, 2020 at 3:56 PM Nicholas Stephanopoulos nicholas.stephanopoulos@gmail.com wrote:

Hi Zhenghong,

Nice to hear from you again! I'm delighted that my spatial diversity work was useful, and I'm looking forward to reading your thesis. I love the idea of looking at how different redistricting criteria are related to one another in hypothetical maps. Hope you're managing okay in this crazy time.

On Thu, May 7, 2020 at 10:41 AM Lieu Zheng Hong lieuzhenghong@gmail.com wrote:

Dear Prof Stephanopoulos,

I hope you and your loved ones are well during these unprecedented times.

I have recently submitted my undergraduate thesis. In it I examine whether or not there is a trade-off between compactness and communities of interest when drawing districts, both of which are mandated by legislatures in the U.S. I operationalise communities of interest with your measure of spatial diversity. I draw hundreds of thousands of counterfactual plans, meant to represent the set of plans an unbiased districting commission would conceivably draw.

I find the following results:

1. Districts' spatial diversity varies a lot between states (your finding), but not so much within states. Each state occupies only a narrow band in the range of possible spatial diversity scores, no matter how we draw the districts. 2. Small urban districts are usually more spatially diverse. 3. Whether or not there is a trade-off between compactness and spatial diversity depends on the compactness measure we choose (Polsby-Popper, Convex Hull, Reock, etc..). Some compactness measures trade-off spatial diversity with compactness, while others synergise well with compactness.

I've attached it here in case you wanted to take a look. Comments are most welcomed, of course.

Without exaggeration, I couldn't have done it without your measure and your data. The spatial diversity measure forms the bedrock of all analyses---without the measure, there would be no thesis. So please accept my sincere gratitude!

Thanks very much and take care, Zhenghong Lieu

Building a deep learning pipeline + model with Keras and PlaidML

2019-09-17T00:00:00Z

I built and iterated on a deep learning model that replicates the outputs of an existing rule-based system while having 100x faster inference. As a prerequisite to iterating on the model I also built a deep learning pipeline that did:

Data reading and cleaning
Feature extraction and selection
Data normalisation
Model checkpointing and saving

I iterated over 6 major model configurations (training ~50 models overall) and got the model RMSE to fall from 0.23 (simple RNN) to 0.128. This significantly improves upon a baseline model (RMSE = 0.22) but there is still room for improvement.

The pipeline I built will speed up future iteration significantly. I will continue the project with the Oxford Strategy Group (Digital), where I might lead the team of consultants.

(Summer 2019 internship with Inzura)

Motivation

Inzura AI has a proprietary algorithm called the "Driver Profiler", which is a complex rule-based system that measures how drivers drive. It derives quantities like acceleration and velocity from GPS data. It looks up driver position in a road network database to check for speeding (this road network database allowed me to do data analysis on average speeds in another project I did for Inzura).

From all this data it identifies speeding/braking/accelerating/turning events and scores them. For instance a hard acceleration would be scored poorly.

Finally, it aggregates all the events to give a "trip score", which is how safely the driver drove on the trip.

This rule-based system works well but has two limitations. First, the database lookups take a lot of time (latency) and necessitate an always-online device.

Second, most of the code for the driver profiler is written in Python. Richard Jelbert, the CEO and co-founder of Inzura, wanted to deploy a version of the driver profiler on dashcams. However, dashcams are embedded systems that run C/C++.

A deep learning model solves both of these problems in one fell swoop. Once trained, it can run inference completely offline, and there are libraries that compile Keras models to C++. Additionally, offline inference means real-time inference: the model could provide real-time feedback to a driver on his performance, which is something that was impossible before.

Feature selection / extraction

There was a lot of data in every second of the trip, but I restricted myself to values that could be calculated relatively quickly and did not require any database lookups. That meant only things like velocity, acceleration, turn radius, not road type or speed limit.

I eventually settled on four features:

Absolute velocity
Tangential acceleration
Radial acceleration
Angle

Data cleaning

There were anomalies in the data: whitespace issues, null values, wrong values, etc. I had to write code that fixed all of that, which was trivial but tedious. ^[1]

Keras generator

I wrote a Python generator to read the training data into my Keras model.

A generator is a "function which returns an object on which you can call next, such that for every call it returns some value, until it raises a StopIteration exception, signaling that all values have been generated. Such an object is called an iterator."

The big benefit of a generator is that they deal with data one piece at a time, and so we don't have to put the whole dataset into memory. With a small number of trips this isn't a problem---50,000 trips may only take up ~2GB in memory---but will be an issue in the future.

One hiccup: A generator must be infinite in order to work with Keras. That is, when it reaches the end of the training data is must loop back again to the start. This took some effort to code (I discovered this very helpful link too late) but will be very helpful when we start training with millions of trips.

[TODO] code here

Normalisation

[TODO] code here

I wrote some simple code to calculate the mean and variance of each feature. I fed the mean and variances into the Keras generator in order to normalise each feature before it was used to train the model.

This was actually quite a slow process: it took about 30 minutes to run. (This is to be expected as there are ~340 million numbers to add). This is a prime candidate for parallelisation with the Raspberry Pi 4 cluster computer.

Model checkpointing

Checkpointing my model helped to save my progress by constantly saving intermediate models. What you do is you train on the training set and only save a model when the validation loss is lower than the best validation loss so far. That means that one can save the best-performing model without worrying about overfitting.

But of course this meant that my initial train/test split would not work. I had to divide it into train/validation/test with a 40000/5000/5000 split.

Here's what checkpointing looks like:

Epoch 00033: val_loss improved from 0.14147 to 0.14128, saving model to cnn_6.7-33-0.1413.hdf5
Epoch 34/50
200/200 [==============================] - 465s 2s/step - loss: 0.1418 - val_loss: 0.1359

Epoch 00034: val_loss improved from 0.14128 to 0.13595, saving model to cnn_6.7-34-0.1359.hdf5
Epoch 35/50
200/200 [==============================] - 440s 2s/step - loss: 0.1391 - val_loss: 0.1395

Iteration

I started off with 10,000 trips. Each trip has length anywhere between 180-10000+ seconds. I extracted the four features at every time step. Null values were taken to be 0.

The ground truth is six subscores that range from 0 to 1: my models try to minimise the root-mean-squared-error (RMSE) between its predicted subscores and the actual subscores.

Mk I: Simple RNN

model = Sequential()
model.add(SimpleRNN(128,input_shape=(None,4)))
model.add(Dense(6)) # Returns a 6x1 vector: predicting 6 different scores
model.compile(optimizer='adam',loss=root_mean_squared_error)

Train and test loss 0.23. I had to abandon the RNN because the training was too slow. It took 40 minutes to train one epoch of 10,000 trips. This is because each trip had a different length and so I couldn't increase the batch size. We could have padded with 0s, but at the time I didn't want to, and decided to move to CNNs instead.

I thought a CNN architecture might work here because I knew the rule-based system was detecting "events".

Mk II: CNN with linear activation

Train and test loss 0.22---not much difference from the RNN model, but by introducing padding I was able to increase the batch size and thus reduce training time by 10x, to 3-4 minutes per epoch.

model = Sequential()
model.add(Conv1D(filters=30, kernel_size=5, strides=2, padding='same',
				 input_shape=(5000,4)))
model.add(MaxPooling1D(pool_size=2))
model.add(Conv1D(filters=30, kernel_size=5, strides=2, padding='same'))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(64))
model.add(Dense(6))
model.compile(optimizer='adam',loss=root_mean_squared_error)

Mk III: CNN with RELU

Changing the convolutional layers to use RELU rather than linear activation functions made a big difference (thanks Jon Chuang), reducing the loss from 0.23 to 0.20 --- just scraping a win against the naive prediction. (Mk III). Furthermore, it seems like I would get more gains from training with more epochs.

model = Sequential()
model.add(Conv1D(filters=30, kernel_size=5, strides=2,
				 padding='same', activation='relu',
				 input_shape=(5000,4)))
model.add(MaxPooling1D(pool_size=3))
model.add(Conv1D(filters=30, kernel_size=5, strides=2,
				 padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=3))
model.add(Conv1D(filters=30, kernel_size=5, strides=2,
				 padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=3))
model.add(Conv1D(filters=30, kernel_size=5, strides=2,
				 padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=3))
model.add(Conv1D(filters=15, kernel_size=5, strides=2,
				 padding='same', activation='relu'))
model.add(Flatten())
model.add(Dense(64))
model.add(Dense(6))
model.compile(optimizer='adam',loss=root_mean_squared_error)

Mk IV: CNN with many more filters and dropout

(Henceforth, I'll be omitting the code because all CNNs look the same, I was just tweaking the parameter size and adding more layers)

I then moved in a different direction --- instead of going deeper (adding more layers), I went wide (adding more filters). This was Mk. IV with 4 CNN layers and 100/160 filters. Sadly, the model didn’t perform that well. But I learned two things from this:

Training speed was not affected! In other words, adding more filters gives you more power “for free” --- at the cost of memory.
Wide filters do not overfit --- in other words, go hog wild with adding larger layers.

Mk V: CNN with batch normalisation

Emboldened by this, I decided to build Mk. V: exactly like Mk III, but increasing the size of the convolutional layers.

I trained 7 more models, trying a combination of different techniques:

5.1 add dropout layer --- training loss 0.2044, test loss 0.2050
5.2 change max pool layers to have size =2
5.3 change stride size = 1: training and test loss 0.20
5.4 append another Conv1D layer at the end with stride size = 2, training loss 0.2042, test loss also 0.204
5.5 remove pooling layers : almost bricks the computer, huge-ass dense layer (2.6 million parameters): training loss 0.1964, test loss 0.2146
5.6 add back the pooling layers: training loss 0.1977, test loss 0.200
5.7 add batch norm in between Conv1D layers.

I knew that all of these models were underfitting as they were not doing much better than the benchmark. So I decided to use MORE layers.

Mk VI: CNN with MORE layers

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv1d_1 (Conv1D)            (None, 5000, 100)         2100
_________________________________________________________________
conv1d_2 (Conv1D)            (None, 5000, 100)         50100
_________________________________________________________________
max_pooling1d_1 (MaxPooling1 (None, 2500, 100)         0
_________________________________________________________________
conv1d_3 (Conv1D)            (None, 2500, 100)         50100
_________________________________________________________________
conv1d_4 (Conv1D)            (None, 2500, 100)         50100
_________________________________________________________________
max_pooling1d_2 (MaxPooling1 (None, 1250, 100)         0
_________________________________________________________________
conv1d_5 (Conv1D)            (None, 1250, 100)         50100
_________________________________________________________________
conv1d_6 (Conv1D)            (None, 1250, 100)         50100
_________________________________________________________________
max_pooling1d_3 (MaxPooling1 (None, 625, 100)          0
_________________________________________________________________
conv1d_7 (Conv1D)            (None, 625, 100)          50100
_________________________________________________________________
conv1d_8 (Conv1D)            (None, 625, 100)          50100
_________________________________________________________________
max_pooling1d_4 (MaxPooling1 (None, 312, 100)          0
_________________________________________________________________
conv1d_9 (Conv1D)            (None, 312, 100)          50100
_________________________________________________________________
conv1d_10 (Conv1D)           (None, 312, 100)          50100
_________________________________________________________________
max_pooling1d_5 (MaxPooling1 (None, 156, 100)          0
_________________________________________________________________
conv1d_11 (Conv1D)           (None, 156, 15)           7515
_________________________________________________________________
conv1d_12 (Conv1D)           (None, 78, 15)            1140
_________________________________________________________________
dropout_1 (Dropout)          (None, 78, 15)            0
_________________________________________________________________
flatten_1 (Flatten)          (None, 1170)              0
_________________________________________________________________
dense_1 (Dense)              (None, 64)                74944
_________________________________________________________________
dense_2 (Dense)              (None, 6)                 390
=================================================================

Mk 6.1: I removed batch normalisation and added more layers for a total of 12 convolution layers. This gave me the lowest test loss so far. Train/test loss 0.1928/0.1857.

The reason why the test loss is smaller than the training loss is because of dropout. From a StackOverflow post:

One possibility: If you are using dropout regularization layer in your network, it is reasonable that the validation error is smaller than training error. Because usually dropout is activated when training but deactivated when evaluating on the validation set. You get a more smooth (usually means better) function in the latter case.

Mk 6.2: I trained for more epochs (40 rather than 20) and got a train/test loss of 0.1698/0.1820. I realised that the model was beginning to overfit and so I got 40,000 more trips.
Mk 6.3: I trained for 50 epochs on 50,000 trips. Training time per epoch took ~20 minutes, which meant the entire training took 15 hours. I got a training loss of 0.163 and a test loss of 0.1599. This is good news --- adding more data stopped the model from overfitting.
Mk 6.4: forgot to write what I did
Mk 6.5: Instead of batch normalisation, I normalised all the inputs and trained the same model again. This really helped! The neural network was able to achieve the lowest-ever training and test loss. Training loss: 0.1475. Test loss: 0.1280
Mk 6.6: removed dropout layer and trained for 25 epochs.
Mk 6.7: Instead of stopping the training at a specific number of epochs, I decided to start checkpointing, and continuously train until the validation loss plateaued. In this way I could plot a learning curve and determine whether I needed more data or a more powerful model.

Here I made silly embarassing mistake #2. I fixed it but didn't have the time to rerun the training.

I stopped the deep learning project here to focus on the Raspberry Pi cluster project.

Silly, embarassing mistakes I made

Not taking advantage of multiprocessing

In the predict_generator function there are several arguments you can pass it to enable multi-CPU operation. One argument was multiprocessing=True which I did set. But what I had neglected to change was workers, which defaults to 1 if unspecified. Once I set workers=10 we got---predictably---a 10x speedup in training.

Using the same data for both the validation and test set

A silly mistake I made with list slicing meant that my validation set was the same as my test set. I saw the validation loss keep decreasing to 0.09 and was incredibly pleased; I thought it was too good to be true. It was!

My thoughts

Before I started this project I had a naive idea of deep learning. I was implicitly assuming the following:

Nice clean-ish data
Nice environment to do deep learning already set up for you
Nice well-defined requirements and unambiguous loss metric

None of those assumptions held true in this project. In particular I had to set up CUDA on my machine (to enable GPU acceleration) and that was really difficult. Eventually I used PlaidML instead of Tensorflow for the backend because it played nice with my GPU.

There's so much glue code that goes into a "business" machine learning model. The actual Keras machine learning code that specifies the model is only a couple dozen lines, but I have hundreds of lines of code that clean the data, normalise the data, choose the right subset of the data, read the data, save the model, run predictions with that model, ... and so on.

Finally, there are many insidious bugs that can bite you silently in deep learning projects. This is probably due to my relative inexperience---I hope that by documenting the silly mistakes here, I'll not make them again.

Things I've learned

This was the first non-toy deep learning project I have done. Because there was no senior data scientist to guide me, I made many silly mistakes and didn't follow best practices.

The most novel part for me was learning about, and writing, the architecture that supports the deep learning model.

Input normalisation improves training significantly. I saw the training/test loss fall from 0.16 to 0.13 just by normalising the input.

In the course of the project I went through some of the courses in Andrew Ng's Deep Learning Specialisation. I didn't find the specialisation very useful but it was good in giving me some deep learning intuitions.

Conclusion

Although I didn't get to finish the project, I still learned a lot. I had to touch every single part of the deep learning pipeline---not just the model code itself---which I really appreciated. A big thanks to Richard for giving me this opportunity.

I watched this Youtube talk on a probabilistic programming language that can infer missing data using Bayesian techniques. This is definitely something I want to explore. ↩︎

Building an Bayesian SMS sender

2019-09-16T00:00:00Z

I architected and built a system that programmatically sends SMSes and tracks each one to see if it has been clicked. I used the pipeline to conduct a behavioural economics experiment on ~2000 users to investigate the effect of loss-aversive framing on clickthrough rates. I used Bayesian statistics (Thompson sampling) to maximise the number of people who were reached by the optimal SMS.

I also sent various other SMSes (a total of ~10,000 messages). Overall, my interventions increased the number of active users by ~25% over a period of two months.

(Summer 2019 internship with Inzura)

As far as I am aware, this is the first behavioural economics experiment that uses Thompson sampling. There are very good reasons why most experiments don't (I elaborate upon this later), but I argue that the multi-armed bandit approach I used had several advantages over traditional A/B testing.

I had previously written a more technical post on Bayesian inference and Thompson sampling. Those interested in the details (or interested in picking apart my misunderstandings) are most welcome to peruse it. This post will talk about why I built the pipeline and the architecture of the pipeline, not the maths.

I have deliberately been vague about the details of the policy and campaign, to safeguard the privacy of our client.

Motivation

Our client is a large insurer who had rolled out a new policy that used our app. If the policyholder uses the app to record his driving (and fulfills some other conditions), he will be eligible for a cashback. However, people were not installing the app. Our working hypothesis was that people had either forgotten about the app, or had forgotten that they had to install it/use it to be eligible for a cashback. Therefore, we wanted to send reminder SMSes to these customers and see if we could get more uptake of the app.

As we were going into this blind, we wondered what sort of SMSes would be most effective. We decided to send three SMSes---a neutral reminder SMS, a "positive" SMS, and a "negative" SMS

Our purpose was twofold: i) to get as many people to install the app, and ii) to see which type of SMS was the most effective.

Why three types of SMSes?

Behavioural economics predicts that people are loss-averse: that is, they feel the pain of a loss more acutely than they feel the pleasure of an equivalent gain. My hypothesis was that an SMS worded to trigger loss aversion ("Install the app now or lose the cashback you're eligible for") would be more effective than an SMS worded positively ("Install the app now to gain a cashback"), which would in turn be more effective than a neutral reminder SMS ("Install the app now").

Why frame the experiment as a multi-armed bandit?

The gold standard of causal inference is to do a randomised controlled trial: randomly divide, say, 2400 participants into groups of 800 each, then send 800 the neutral (control), 800 the positive, and 800 the negative. Record the number of successes (clicked SMS) and failures (did not click SMS) for each treatment.

This is Good and Rigorous™. In particular, it maximises our certainty about the performance of each treatment. In most experimental contexts, this is fine. Most experiments try to uncover some fact about the world, and to maximise our certainty in the truth of those facts. But there are three additional considerations with this experiment design when it comes to the real world.

The first and biggest consideration is that nobody is going to let you run it because this is leaving money on the table. With this design 2/3rds (1600) of the customers will be getting a suboptimal SMS. In a business environment, we want to maximise the number of customers who receive the optimal SMS, not really maximise the certainty of our knowledge.

The second consideration is that we don't really want to be certain about the performance of every treatment. We want to be certain of the best treatment, which means that a uniform allocation (800 SMSes to each group) is nonideal. ^[1]

The third consideration is that we don't have to send all the SMSes at the same time. We can send a subset of the SMSes and use that initial data to draw tentative conclusions about which SMS is the best.

Given that we want to maximise the number of customers who receive the optimal SMS, the traditional experimental design is nonoptimal. But we don't know which SMS is best, so how do we find out as quickly as possible, and then maximise the number of customers who receive it?

It turns out that this problem can be neatly modeled as a "multi-armed bandit":

You are walking around the casino floor where there are ten slot machines, each with a different probability of paying out. How should one pull the slot machines in order to maximise one’s payout over the long run?

In this case, we have three slot machines (neutral, positive and negative SMSes). And since we don't know which arm is best, we have to send some SMSes initially to find out.

But there's a balance to be struck here. We want to spend some time learning, but after we're "pretty sure" about what the best arm is, we should keep pulling the best arm all the time. This is known as the explore vs exploit tradeoff (or, more pithily, "learn or earn"). We can see now why the initial experiment design was not fit for purpose: it's very good at learning, but no good at earning. (And we're trying to earn money here!)

Why Thompson sampling?

There are many proposed algorithms to solve the multi-armed bandit problem: these include epsilon-greedy, upper confidence bounds (UCB) and Thompson sampling.

I spent about a week reading the literature and thinking about it and eventually decided on Thompson sampling. More details are covered in my more technical post.

Experiment design

I decided on the following three SMSes (neutral, positive and negative):

You have not logged any trips. The <POLICY> you have purchased requires you to log trips regularly on the <APP> app. Install the app and start logging your trips today. Any problems, call <NUM>, or email <EMAIL>. Click here to start: http://<*>.inz.ai/<SMS_ID>

You have not logged any trips. The <POLICY> you have purchased requires you to log trips regularly on the <APP> app. Install the app and start logging your trips today to be eligible for your (up to) <AMT> cashback! Any problems, call <NUM>, or email <EMAIL>. Click here to start: http://<*>.inz.ai/<SMS_ID>

You have not logged any trips. The <POLICY> you have purchased requires you to log trips regularly on the <APP> app. Install the app and start logging your trips today or you will lose your (up to) <AMT> cashback! Any problems, call <NUM>, or email <EMAIL>. Click here to start: http://<*>.inz.ai/<SMS_ID>

While traditional multi-armed bandits allow you to pull one arm at a time and wait for the results, we didn't have the luxury of time here. Here we have to send thousands of SMSes in a short period of time, and we can't wait for the result of an SMS to come in (a user might take days to click an SMS if at all) before sending the next one. The simplest way is to simply send more SMSes in one go, but I wasn't sure if this would affect any of the nice guarantees (e.g. regret bounds) of Thompson sampling. I tried asking StackExchange but got no response.

I had to push through in any case. The modified Thompson sampling procedure works as follows. I start by sending a total of 300 SMSes per batch: in expectation, 100/100/100 of each type. Some of these SMSes will be clicked on and some will not.

Once the results come in, the algorithm will preferentially choose to send more of the better-performing SMS and fewer of the worst-performing one. For instance, after the first round the algorithm might send 124/80/96.

As more batches are sent, the algorithm builds up a better and better picture of the relative efficacy of each SMS, and will converge upon sending the best SMS exclusively.

In this experiment I sent 6 batches of SMSes.

Architecture

I knew what SMSes I wanted to send; I now had to build a system that would allow me to send it.

There are four key parts of this pipeline:

Inzura server that contained customer data
NGINX server to log clicked SMSes
PostgreSQL database to log customer data and status of all SMSes sent
Multi-armed-bandit code to calculate the optimal number of each SMS to send

I logged clicked SMSes in the following way:

SMS tracking architecture

Step 1: The most important part is to measure the effectiveness of each SMS variant (neutral, positive or negative). I generate unique SMS IDs (3 alphanumeric characters + checksum) for each SMS and append them to a template string, making a unique SMS string. I then make GET requests to the service that sends SMSes, the Ant API.

Step 2 and 3: The user receives an unique SMS with a link at the end. Upon clicking it, the user is taken to a logging server on the inz.ai domain, which logs the ID of the SMS and redirects the user to the app store.

Step 4: I download the server's logs and process it, and update my database of which SMSes have been clicked.

Data analysis and MAB architecture

Now that I had a system to log SMS clicks, I had to implement the experiment.

First I had to create a PostgreSQL database and decide upon the schema for it.

Secondly, I had to pull customer data from the Inzura server and add it to the PostgreSQL database to identify eligible customers (we obviously don't want to send reminder SMSes to customers who have already installed the app).

Third, I had to write the Thompson sampling code that would read the data from the database (the slot machine labeled "MAB"). It calculates the optimal number of each SMS to send given the SMSes' past performances.

Finally, we generate unique SMSes and give them to the Ant API, and the cycle restarts. I call this a "pipeloop" because it's a pipeline that loops upon itself.

Thompson sampling code

This is the code that performs Thompson sampling. First, some definition:

Each arm is a tuple of the number of clicks and no-clicks of each SMS plus one. For instance, if I had sent 100 SMSes and there were 7 clicks, then the arm would be (8, 94).

The code takes a tuple of arms e.g. [(8, 94), (15, 87), (2, 103)] and the number of samples we want to take (ergo the number of SMSes we want to send this batch). It will then give us the optimal number of each SMS to send. In the example above, the algorithm will tell us to send more of the 2nd SMS, some of the 1st SMS, and almost none of the 3rd.

'''
Thompson sampling code
'''
from collections import namedtuple
import numpy as np
import random
from iteround import saferound

Arm = namedtuple('Arm', 'alpha beta')


def thompson_sample(arms, N):
'''
Takes as input arms and number of pulls we need to make, and
returns a tuple of arms to pull
'''
    samples = []
    arms_to_pull = []
    for n in range(N):
        samples.append([])
        for arm in (arms):
            samples[-1].append(sample(arm))
        # pick the best arm
        arm_to_pull = samples[-1].index(max(samples[-1]))
        arms_to_pull.append(arm_to_pull)
    return arms_to_pull

def sample(arm):
	return np.random.beta(arm.alpha, arm.beta)

Results

I got a null result. There wasn't much of a difference between loss-aversive SMSes (negative framing), positive SMSes and neutral SMSes.

Because we are using Bayesian statistics here, we can calculate the probability that loss-aversive SMSes are more effective than neutral or positive ones directly.

I calculated the following using a simulation approach:

There is a 26.6% probability that loss-aversive SMSes are more effective than neutral SMSes.

There is a 60.7% probability that loss-aversive SMSes are more effective than positive SMSes.

Obviously none of these results are very credible.

Code for the simulation:

import random
import numpy as np

def sample(tup, N):
    return (np.random.beta(tup[0], tup[1], size=N))

N = 1000000

neutral = sample((69, 828), N)
positive = sample((10, 154), N)
negative = sample((19, 266), N)

c = list(zip(negative, neutral))
c = list(zip(negative, positive))
d = [1 for x in c if x[0] > x[1]]
print(sum(d) / N)

The following diagrams are beta distributions. Think of them as a measure of our uncertainty in the effectiveness of each SMS. The x-axis measures the effectiveness of each SMS, and the y-axis measures our degree of certainty. The higher and sharper the distribution, the more certain we are about the effectiveness of an SMS.

The diagrams show the evolution of the estimates over time by SMS variant: red is neutral, green is positive, and blue is negative. Overall we can see that the distributions lie very closely to one another.

When we begin, we have no idea how effective the SMSes are, so we assume that all SMSes form a uniform distribution. ^[2]

As we send SMSes and get the results back we begin to update our beliefs about the SMSes. Here the neutral SMSes seem to do best and thus the Thompson sampling algorithm chooses to send a lot of it.

But the good performance of the neutral SMSes is quickly revealed to be a fluke as more SMSes are sent out.

Eventually after all the SMSes have been sent we realise that all of the SMSes are about the same in effectiveness.

Due to time constraints, I could not run further experiments: I did A/B testing henceforth sending multiple reminder messages and again got null results. So I'm glad this was a very robust null finding :P

Nonetheless, my further interventions were able to increase the number of active users by around ~25%. I will not reveal numbers here for privacy reasons, but this had a real business impact because Inzura gets paid per monthly active user.

What I learned

I gained valuable practice in designing an experiment and carrying it to completion. I had originally wanted to kill two birds with one stone and use this for my BEE thesis, but a null finding isn't particularly exciting, so I will keep looking. ^[3]

I learned quite a lot about Bayesian inference in the course of designing the experiment, and would like to learn more about it. When I have more bandwidth I will consider doing the Probabilistic Graphical Models MOOC by Daphne Koller, and reading Judea Pearl's co-authored book on DAGs (highly recommended by Bassel).

I had to learn SQL in order to build the databases. In the course of doing this project I worked through the Stanford Databases MOOC by Jennifer Widom. I completed DB1 Introduction and Relational Databases, DB4 Relational Algebra, DB5 SQL and DB8 Relational Design Theory. DB4 and DB8 helped me design the schema and DB5 improved my very hackish/amateur queries into much more elegant and readable ones.

Lastly, I was able to stretch my mind by architecting a complex system. It's kind of like thinking about the Lego parts you'll need to 3D-print to build the rocket ship you have in mind. It was a good exercise to think about what parts I needed in the system and how they would talk to each other. (I guess this is what senior SWEs like Chris and David do in their sleep.) I've greatly simplified the architecture in my breezy explanation---the actual architecture I built has more moving parts, and it was quite challenging for me.

Conclusion

I am very glad that my knowledge of statistics and behavioural economics could be brought to bear in this project. I have always gone through my degree thinking that it was useless for doing anything technical.

This project gave me an idea about my comparative advantage in data science. While it isn't difficult for a computer science major to learn statistics, many of them might not have done so. If I max out my statistics and maintain my ability to write software I could be some sort data-scientist data-engineer hybrid; not a "pure" data scientist but someone who likes to get his hands dirty wrangling the code and building the systems.

This is the problem of "best arm identification". For N > 2, it isn't optimal to pull all arms equally. If one arm is very obviously worse than the rest, the algorithm shouldn't waste time on pulling it more. this paper (Best Arm Identification in Multi-Armed Bandits) has more details. See also this paper on Heoffding and Bernstein races where the objective is to try to establish the best action at a given confidence level with the fewest pulls. ↩︎
The choice of a uniform prior was because I had never run an experiment like this before, and had no idea what the clickthrough rate of an SMS was supposed to be. For subsequent experiments I would initialise the priors at around a 0.07 click rate. ↩︎
Yes, I know I'm contributing to publication bias---but my grades are on the line here... ↩︎

Doing distributed data analysis on a Raspberry Pi cluster

2019-09-12T00:00:00Z

(Summer 2019 internship with Inzura)

This is the second project I've done with Raspberry Pis. The first one was a Raspberry Pi game console done back in 2017.

As far as I know, this is the largest Apache Spark Raspi cluster that has been published online. There's an eight-node Raspi 4 cluster here and a six-node Raspi 2 cluster here but i) ours is the biggest and ii) we have actually used the cluster to do "real" data analysis.

Results

I calculated average traffic speeds of a sample of 50,000 trips in the UK. These trips were recorded from drivers who used Inzura's mobile app. The vast majority of trips were in Northern Ireland, and I've chosen to zoom in on the traffic data in Belfast, Northern Ireland. These are chloropleth maps: roads with low average speeds are dark purple, and roads with high average speeds bright yellow.

This is a really high-resolution image, best opened and viewed in full-size. It shows the extent of coverage of the trips that I sampled. We can see that while the data covers the entirety of the UK, most of the trips are from Northern Ireland (NI).

As we zoom into NI we see that almost the entire landmass has been traversed. We can clearly make out city centers, and the roads that connect them: Belfast is by far the largest and most prominent---although one can make out other cities like Bangor and Lisburn near it. We can also see Donegal and Omagh in the west.

We can glean some insights even at this distance. We can see that average speeds in cities tend to be low (dark purple), while roads that connect cities have higher average speeds.

Let's zoom in a little:

We can now clearly see the shape of Belfast and the cities near it. There's an interesting funnel shape at the top, almost like it's feeding into Belfast proper. There's also what looks like extensions of the city to the south-west: are these part of Belfast too, or cities in their own right?

From the road network alone we can see that towns usually set up near the coast: the road network on both sides of the bay is much denser than bits closer inland.

Now let's zoom in to Belfast proper:

At this zoom level the layout of the city can clearly be discerned. Streets near the center of the city are laid out in a neat and densely-packed grid pattern, which gives way to a sparser, more organic layout toward the

We can even see some ferry routes from the pier! These are routes to the mainland and the Isle of Man.

Most interestingly, we can see an S-shaped road that snakes through the city. It starts from the southwest and winds its way up northeast. We can see that it is a different colour from the rest of the city's roads, which are mostly dark purple: this road has segments of yellow and green.

Here's the same image with the background darkened so we can see the colours more clearly:

We can really make out the S-shaped road here, and a couple more besides. There's a road leading down from the northeast that looks like a main connection to the northeastern coast.

One final zoom in to 1:25000 scale:

The shape (and colour) of the S-shaped road is apparent now.

Average speeds on the S-shaped road decrease when there is a bend in the road, and when there is an intersection. Notice also that roads on the periphery tend to have higher average velocities, although this could be because there is not enough data on them (thus throwing off the averages).

One final render without the background---just because it looks pretty, and shows us the shape of the city:

Raspberry Pi compute cluster

This is the Raspberry Pi compute cluster. It's taller than a man:

It was designed and built by Richard Jelbert, the CEO and co-founder of Inzura. Almost everything you see is 3D-printed: that includes the faceplates, the clips and the holding rack (for the Raspis). It contains 16 Raspi 4s and 25 assorted Raspi 3B/3/2/1s. For this project I only used the Raspi 4s.

The structure of the cluster is hexagonal: there are 5 storeys and 6 segments on each storey. One of the segments is reserved for SSDs (data storage and database lookups), but the other 5 segments can be used for compute.

I'm sorry Dave, I'm afraid I can't do that

The front panel opens up to expose the internals:

Again, all of this is 3D-printed. 8 Raspberry Pi 4s sit on each rack and are cooled by 4 mini-fans per rack (the cooling is necessary to prevent them throttling).

The cluster has a theoretical maximum capacity of 400 Raspis (16 Raspis per segment, 5 segments per storey, 5 storeys total); Richard said that cooling may be tricky at that density, but 12 Raspis per segment is definitely doable. That gives us an actual capacity of 300 Raspis in the cluster.

It is also possible to cluster MinIO across up to 16 nodes. Minio allows distributed object storage: it lets my code run without having a copy of every single file on every single Raspi. Given that we need only 1 SSD per storey, this will be sufficient for our purposes.

Why use the cluster?

Why go to all the trouble to set up a compute cluster? Why not run it on a single machine?

It's the only scalable solution when the number of trips gets large
We get increased performance by parallelising computation
It's cool!

Scalability

I had only been able to download a small sample of 50,000 anonymised trips. But Inzura has a corpus of millions of trips and growing. It certainly doesn't fit on a single machine's memory (it doesn't even fit on disk). So the most naive approach of reading all the files into memory at once and calculating the average that way won't work. Something that does work is to read the files in one by one and calculate the averages that way, but this is of course rather slow. Which leads us to ...

Performance

The task is to calculate average road speed for every road in the database. This is an embarassingly parallelisable task: each trip is an individual JSON file (which means we don't have to worry about shared memory), and there is almost no work needed to combine the results of disparate parallel computations. This means that we can get a close-to-linear speedup simply by adding more computers --- which means that a cluster of Raspberry Pis will outperform even a beefy (and much more expensive) computer.

Even if 16 Raspis don't outperform a single computer, 300 Raspis certainly will, and it is easy now to add more Raspis now that the groundwork has been laid. This opens the door to all sorts of future data analysis and exploration tasks which may be prohibitively expensive for a single computer to run (in terms of wall-clock time).

Cool factor

Why not just use Google Cloud or something?---because this is cooler and more educational.

Configuring the cluster

Configuring the cluster was by far the hardest part of this project. Any instructions I could find for setting up a Spark cluster were either incomplete, out of date, or incorrect. It took Richard, Chris and I three full days to set everything up. Eventually I heavily modified this MinIO cookbook and successfully deployed Spark on the 16 worker Raspis:

The cluster computer has 64 cores and 64GB of memory (45GB free)---already much better than my laptop.

I have written a bash script that should automate (but not completely) the installation process here. The only thing one has to do apart from this is to edit the core-site.xml file to point to one's S3 bucket/MinIO server.

Apache Spark code

I gained the necessary knowledge to do this data analysis project by working through part of the Functional Programming in Scala Specialisation. I finished the courses Functional Programming Principles in Scala, Parallel Programming and Big Data Analysis with Scala and Spark. The last course was the most directly relevant to this project but I found the first course incredibly helpful for learning Scala (and the functional programming paradigm), and the third course for reasoning about parallel programs. Both courses helped me appreciate why Scala is a good language for data analysis---many of the functional abstractions of mapping over an iterable of some sort carry over almost directly to distributed computing.

I found Spark quite concise, although the documentation was quite lacking. Nonetheless, I was able to write the program in only about 20 lines of code.

The code does the following:

Read a JSON file onto the Raspi
Extract all the roads and velocities from the JSON file (Map step)
Combine all the roads and find the average velocities from all the JSON files (Reduce step)
Saves it into a single CSV file (the coalesce step). This step is optional (and in fact is probably not recommended when the data set gets larger). I did it to make it easier to visualise in QGIS.

In total there were 50,000 trips, ~85 million data points, and ~170,000 unique road IDs.

object SimpleApp {
  def main(args: Array[String]) {
    import org.apache.spark.sql.SparkSession
    import org.apache.spark.sql.functions.explode
    val spark = SparkSession.builder.appName("TripAnalysis").getOrCreate()
    import spark.implicits._
    val results_path = "s3a://results/"
    val paths = "s3a://trips/*"
    val tripDF = spark.read.option("multiline", "true").json(paths)
	// "Explode" the data array into individual rows
    val linksDF = tripDF.select(explode($"data").as("data"))
    val linksDF2 = linksDF.select("data.dbResponse.linkID", "data.absVelocity")
    // create a temporary view using the DataFrame
    linksDF2.createOrReplaceTempView("times")
    /*
      root
      |-- linkID: string (nullable = true)
      |-- absVelocity: double (nullable = true)
    */
    val tDF = spark.sql("SELECT CAST(linkID as LONG), absVelocity from times
	WHERE linkID IS NOT NULL AND absVelocity IS NOT NULL")
    val groupedDS = tDF.groupBy("linkID")
    val avgsDS = groupedDS.agg(
      "linkID" -> "count",
      "absVelocity" -> "avg"
    ).sort($"linkID".asc)

    avgsDS.coalesce(1).write.
    option("header", "true").
    csv(results_path + "results_49998")
    spark.stop()
  }
}

The Spark code gives the road IDs, but we still have to get the coordinates that that road ID corresponds to. So I queried the road database (which also runs in the cluster) with a couple of lines of throwaway Python code.

I had written a SQL utility function for my previous project on optimising SMS messages which I was able to reuse here.

import time
from sql_utils import query

# First read csv

import csv

rows = []

with open("results-49998.csv") as f:
    csvreader = csv.reader(f)
    header = next(csvreader, None)
    print(header)
    for row in csvreader:
        rows.append(row)

BATCH_SIZE = 10000
NUM_ROWS = len(rows)

pointer = 0

# Break up the rows

link_ids = [x[0] for x in rows]

all_start = time.time()
while pointer < NUM_ROWS:
    start = time.time()
    link_ids_subset = link_ids[pointer:pointer+BATCH_SIZE]
    array_string = ", ".join(str(i) for i in link_ids_subset)
    array_string = f"({array_string})"
    query_string = f"SELECT geom from (SELECT DISTINCT(link_id), " +
	"geom FROM streets WHERE link_id IN {array_string}) as f"

    # Send out the query
    query_result = (query(query_string))
    for i in range(pointer, min(pointer+BATCH_SIZE, len(rows))):
        rows[i].append(query_result[i-pointer][0])

    end = time.time()
    print(f"Time taken to do {pointer}:{pointer+BATCH_SIZE}: {end-start}")

    pointer += BATCH_SIZE

all_end = time.time()

print(f"Time taken to make all the queries: {all_end-all_start}")

header.append("geom")

with open("results-49998-with-geom.csv", "w") as f:
    f.write(",".join(str(i) for i in header))
    f.write('\n')

with open("results-49998-with-geom.csv", "a") as f:
    for row in rows[:NUM_ROWS]:
        f.write(",".join(str(i) for i in row))
        f.write('\n')

There are issues with such an approach: when the number of unique road IDs get large, we won't be able to load all the rows into memory like I do here. But there were only ~140,000 unique link IDs, and I think even if you were to cover the whole of the UK the number of links would not exceed 2 million --- which fits well within the memory of a single machine.

I made two main optimisations.

First of all, I batched up the queries (instead of querying SELECT * FROM streets WHERE link_ID = linkid, do WHERE link_ID IN {array_string}) because I knew that latency would be the primary bottleneck. This gave a huge speedup: making 10,000 queries (without caching) only took about 1.2 seconds. To give a comparison, measured round-trip time was about 0.1s (100ms); if I had made 140,000 individual queries, this would have taken 14,000 seconds (4 hours!), and I was able to do all the queries in 14 seconds.

Second, I knew that the linkIDs were indexed in the database, and so I made sure to sort the linkIDs before batching and sending them to maximise the probability of a cache hit.

(base) lieu@bigbeast:~/dev/cluster-computing$ python3 get_road_geom.py
['linkID', 'count(linkID)', 'avg(absVelocity)']
Time taken to do 0:10000: 0.5610289573669434
Time taken to do 10000:20000: 0.48032593727111816
Time taken to do 20000:30000: 0.5004429817199707
Time taken to do 30000:40000: 0.5266928672790527
Time taken to do 40000:50000: 0.5340712070465088
Time taken to do 50000:60000: 0.6023671627044678
Time taken to do 60000:70000: 0.5589091777801514
Time taken to do 70000:80000: 0.48449015617370605
Time taken to do 80000:90000: 0.543027400970459
Time taken to do 90000:100000: 0.5296523571014404
Time taken to do 100000:110000: 0.6949601173400879
Time taken to do 110000:120000: 1.437162160873413
Time taken to do 120000:130000: 1.2379283905029297
Time taken to do 130000:140000: 1.15696120262146
Time taken to make all the queries: 9.848754644393921

The final CSV looked like this:

linkID;count(linkID);avg(absVelocity);geom
17341260;3;25.53091932029027;LINESTRING (-4.17935 53.21241, -4.17968 53.21257, -4.18033 53.21294, -4.18115 53.21319, -4.18138 53.2133, -4.18184 53.21339, -4.18204 53.21346, -4.18262 53.21369, -4.18334 53.21389, -4.18394 53.21439, -4.18473 53.21465, -4.18535 53.21475, -4.18618 53.21478)
17341303;1;31.053999640650744;LINESTRING (-4.17386 53.2051, -4.17501 53.20569, -4.17523 53.2058, -4.17534 53.20584)

The LINESTRING is a Well-Known Text (WKT) geometry representation and corresponds to the coordinates that a road occupies.

Future extensions

Now that the groundwork is done, future extensions become merely an issue of writing new code in Spark (which is, IMO, the easiest part). Some very nice and (relatively) low-hanging fruit to pluck include:

Subdividing average speeds by time of day (so we can see how traffic conditions vary with time). We can even do a nice animation.
Bringing in the full database of millions of anonymised trips in the corpus (possibly by connecting directly to the AWS servers rather than using a local MinIO instance)

Additionally, we can use the calculated average speeds to give a more personalised metric of driver safety. At the moment, a driver's score is determined (in part) by whether or not he drives over the speed limit of a road. But many roads do not fit their posted speed limits. It would be more enlightening to compare a driver's driving patterns to a reference distribution of other drivers on the same roads---and this data analysis helps us create that reference distribution.

Finally, we could use the ground truth of average speeds as input to a machine learning model. Can we predict the probability of a driver getting into an accident given the roads that he drives on, the speed he drives on those roads, and how he drives on them? A reference distribution is surely instructive here: it's not as dangerous to go fast when everyone else around you is going fast, but it's very dangerous to go fast when everyone else is slow (or go slow when everyone else is fast).

From Richard:

I think big data analysis is a definite. GPUs will be faster than CPU at certain simple tasks so worth keeping to tasks that distribute well across CPUs (small input data, lots of complex math, small output data / results). Inference technology is also getting cheaper and dedicated neural network hardware could be added to each node. Like the Google fpga or Intel hardware. this may be faster than GPU

you should mention that Spark was bare metal and that the next step will be to migrate to Docker Swarm deployed Spark nodes

What I learned

Scala, Spark and distributed computing were completely new to me, and I had a great time learning them. Learning about the abstractions of functional programming expanded my mind. For instance, I had heard about monoids and monads before but I only now understand their significance. Something that really clicked for me was an explanation of how monoids map easily to parallel programming, due to their associativity and identity.

Spark was cool as well. There were many helpful tips in the course about optimising one's Spark program (always try to use Pair RDDs/Datasets, avoid shuffles whenever possible, minimise the data sent over the network, using range partitioning...) but sadly none of them were useful for this project.

It was difficult trying to get Spark working on the Raspberry Pi cluster---in part because not many people have done it---but I'm pleased to have cracked this tough nut. In fact the project made me want to build my own Raspi cluster...

Conclusion

I am very happy to have had the opportunity to work on this project. I am very grateful to Richard Jelbert for giving me this very cool project to work on.

Richard seems to be glad too. Running data analysis on the cluster with Spark has been on his wishlist for quite a while now, but nobody has had the bandwidth to do it.

The beta-Bernoulli bandit, Thompson sampling, and Bayesian inference

2019-07-12T00:00:00Z

I'm using the work I do at my internship to run a behavioural economics experiment. Trying to run this experiment has led me down an incredibly deep rabbit hole, and I wanted to write down some of the things I've learned before I inevitably forget.

I also wanted to use this blog post to defend many of the decision points I made---there were many points where I had to make a decision to use one approach or another, and I spent many hours thinking and reading about the best approach.

Background

I'm trying to send SMSes to customers telling them to install an app. Behavioural economics predicts that people are loss-averse, so I am trying to find out if sending SMSes that have a loss-aversion framing

"install the app now, or you'll lose your 4500THB cashback"

has a greater effect than sending SMSes that have a positive framing

"install the app now, to get 4500THB cashback"

The question is:

How can we find out which SMS is best?

The gold standard of causal inference is to do a randomised experiment: randomly divide my 2400 participants into groups of 800 each, then send 800 the neutral (control), 800 the positive, and 800 the negative. Record the number of successes (clicked SMS) and failures (did not click SMS) for each treatment.

By the Central Limit Theorem all of these will be normal distributions. We can then do the usual t-tests/chi-squared test and test that there is a statistically significant difference between the means. (More precisely, we reject the null hypothesis that there is no difference between the means).

This is Good and Rigorous™ — but the problem is nobody is going to let you run it because this is leaving money on the table. Only 800 customers are going to get the optimal treatment, but that means 2/3rds of them will be getting a suboptimal treatment.

Is there a better way? Specifically, is there an optimal way to send the SMSes so that we can "know" which treatment is best with some notion of certainty, yet still get the most number of people receiving the optimal treatment at the end of the day?

Why this can be modelled as a multi-armed bandit problem

It turns out that this is an incredibly-well-studied field, with a lot of applications everywhere from medical trials to (of course) marketing.

The explanation of the multi-armed bandit is incredibly intuitive, and it's something you just grok the instant you hear about it.

You are walking around the casino floor where there are ten slot machines, each with a different probability of paying out. How should one pull the slot machines in order to maximise one's payout over the long run?

There are two tradeoffs here. On the one hand, we want to explore in order to find out what the best slot machine is. The best way to be sure of what the actual probabilities of the payouts are is to pull every arm equally.

But on the other hand, do we really care about what the actual probabilities of all the payouts are? Does it really matter if slot machine #4 could have a probability of 0.01--0.03 if we are pretty sure that slot machine #5 has a probability of 0.40-0.60? Fundamentally we want to exploit: if we are already 90% sure that this slot machine is the best one, we shouldn't keep pulling the others to "explore", because that will in expectation reduce our earnings. This post was incredibly helpful for getting the intuitions, and I'll refer to this post again and again.

In the same way, it's suboptimal ^[1] to run a "pure exploration strategy" of 800/800/800 split. Once we've made, say, 100 pulls each, we should be more or less "sure enough" about the probability of each treatment, and therefore we should "shift gears" and start the exploitation.

OK, so now I've shown that if we want to maximise our payout (also called the "cumulative reward"), then a pure exploratory strategy is suboptimal. That leads to a very natural follow-up question:

What is the optimal strategy to maximise cumulative reward?

Now that we have intuition firmly in our heads, it is useful at this point to introduce some mathematics so that we can more formally specify the problem.

I'll not focus here on the general multi-armed bandit (MAB) problem. Instead I'll specify the Bernoulli bandit, and I'll modify slightly the description from this paper

(Bernoulli Bandit) Suppose there are $K$ actions and $T$ time-steps, and when an action is played, it yields either a success or a failure. Action $k \in {1, ..., K}$ produces a success with probability $\theta_k \in [0,1]$ . If there is a success, then $r_t = 1$ , else $r_t = 0$ . The success probabilities are unknown to the agent, but are fixed over time. The objective is to maximise the cumulative reward $\sum^T_{t=1} r_t$ over $T$ periods, where $T$ is relatively large compared to the number of arms $K$ .

Let's first think about a few simple strategies. Let's start with the worst:

Random -- choose an arm at random every period.

First of all, the random strategy is exactly that---random---and the expected cumulative reward for this strategy is simply

\sum^K_{k=1} \frac{T}{3} \theta_k.

For instance, if $\theta = \{0.3, 0.4, 0.5\}$ and $T=300$ , then the random strategy would give $\mathbb{E}[r] = 30 + 40 + 50 = 120$ .

Notice that the Random strategy is exactly the "random assignment" algorithm of assigning 800/800/800.

Can we do better? Let's try the following:

"Dumb" Greedy(N) -- pull each arm N times. Pick the arm with the highest sampled $\theta$ ; call it arm $K^{*}$ . Then always pull arm $K^{*}$ henceforth.

Note the two degenerate cases: if $N=0$ , Dumb Greedy picks an arm at random and pulls it all the way. If $N=T/K$ , the greedy approach reduces to the random strategy. For intermediate values of N, we first pull all arms N times, to get a better idea of the distribution, then checks the best arm and pulls that. Greedy(N) dominates random choice, because it gets some information about which arm is best, and uses that information to inform its choice.

This strategy's effectiveness follows a U-shaped curve: it's no better than random choice if you just pick a random arm at t=0 and pull it forever, or forever pick a random arm. By the law of large numbers, the more you pull, the more certain you are about the true means of the arms, but the less time you have to pull the best arm. This captures our intuition of exploring and exploiting: the knowledge you gain from exploring is good, but only if you manage to exploit that knowledge.

TODO (questions for author): How much better is Greedy(N) than random? What is the optimal N?

It would be good if we had some measure of not just the mean, but also how certain we were of our estimate of the mean. If I flip a coin 10 times and get 1 head and 9 tails, I am much less certain of its fairness as if I were to flip that coin 500 times and get 50 heads and 450 tails.

It turns out that measure does exist, and it's called ...

The beta distribution

The beta distribution can best be thought of as "a distribution of probabilities --- that is, it represents all the possible values of a probability when we don't know what that probability is."

This Stats.SE thread entitled "What is the intuition behind beta distribution" has loads of fantastic answers, and it's where I got that explanation from. I would certainly recommend reading through David Robinson's answer.

Here's how I would explain it:

We know that the treatments are Bernoulli with parameter $p$ . Then in $N$ repeated trials the treatments must follow a binomial distribution with $B(N, p)$ .

Suppose we conduct N trials and observe $\alpha$ successes and $\beta$ failures. Then the beta distribution $Beta(\alpha+1, \beta+1)$ gives us the range of "reasonable estimates" of the binomial distribution parameter $p$ .

This image by Raffael Vogler very nicely shows why a beta distribution gives us what we want. We wanted a measure of the mean but also a measure of our certainty that the mean is what we think it is. Here we can see that while the orange curve has a higher mean (0.9), our "best estimate" of its true value is quite diffuse. On the other hand, we're quite sure that true value of the second curve (in yellow) lies close to 0.8.

Furthermore, the beta distribution has a lovely property in that we can "update" the beta distribution very easily as we get more data. Suppose we have 2 heads and 8 tails: the beta distribution is $Beta(3, 9)$ . We flip the coin again. It comes up heads. Then the beta distribution easily updates to $Beta(4,10)$ .

Armed with this knowledge of beta-distributions, we can try the following strategy.

Can we do better? Let's try the following:

"Smart" beta-Greedy: At every time step, calculate $\hat{\theta}$ of every arm, then pull the arm with the highest $\hat{\theta}$ and update the $\hat{\theta}$ according to the result.

How "Smart" beta-Greedy works: Initialise each arm with $\hat{\theta} = \mathbb{E}[Beta(1, 1)] = 0.5$ . The $Beta(1,1)$ distribution is a uniform distribution over [0,1], and is what's known as an uninformative prior: we assume that all rewards are equally likely.

At the start, seeing that all thetas are equal, we pull an arm at random; call this arm $k$ . We update $\hat{\theta_k}$ appropriately.

We update as follows:

If we do get a reward: $(\alpha_k, \beta_k) = (\alpha_k+1, \beta)$
If we do not get a reward: $(\alpha_k, \beta_k) = (\alpha_k, \beta+1)$

Finally, pick the best $\hat{\theta}$ , which is given by $\frac{\alpha_k}{(\alpha_k + \beta_k}$ . Repeat this process.

This greedy algorithm is smarter than "dumb" greedy, because it constantly reevaluates what the best arm is with our new beta updating rule. This is pretty good, right?

Can we do better?

Take a look at this image from A Tutorial on Thompson Sampling.

There are three possible arms. The greedy algorithm would always choose action 1, because action 1 has the maximal expected mean reward. "Since the uncertainty around this expected mean reward is small, observations are unlikely to change the expectation substantially, and therefore, action 1 is likely to be selected ad ininitum."

It seems reasonable to avoid action 2, since it is extremely unlikely that $\theta_2 > \theta_1$ , but the algorithm should try out $\theta_3$ , because if you take a look at that distribution in red it's quite plausible that its mean reward exceeds 0.6. "The algorithm fails to account for uncertainty in the mean reward of action 3, which should entice the agent to explore and learn about that action." (p. 10-11)

Let's try and capture some intuition here. The problem with "Smart" Greedy is that it can be misled by an initial lucky or unlucky streak, and get stuck in a suboptimal maxima of exploitation. We could modify "Smart" Greedy to pull every arm N times first, just like "Dumb" Greedy(N), but here's another way to do it:

ε-Greedy -- like "Smart" Greedy, but with probability $\epsilon$ pull an arm at random.

The ε-greedy algorithm takes the best action most of the time, but does random exploration occasionally. This will prevent the algorithm from getting "stuck" on a local optima: 5% of the time it will try action 3, for instance, and learn more about it. Eventually the ε-greedy algorithm will converge onto the best action.

TODO (questions for author): How good is ε-Greedy compared to Smart Greedy?

But again ε-greedy is not optimal for two reasons. First, if we're already dead certain about the best lever, we don't want it to pull a random bad lever ε of the time! Second of all, it wastes pulls exploring arms we're already certain about. In this case we want to explore action 3, and we don't want to explore action 1 at all. Yet the ε-greedy algorithm will select action 1 and 3 with equal probability.

This is suboptimal. We need something that intelligently allocates exploratory effort: go where your estimates are the most uncertain. Can we do better? Enter...

Thompson sampling

Thompson sampling is a very subtle modification of the "smart" greedy algorithm: so subtle, in fact, that I missed it the first time! Here's the difference: instead of choosing the highest expected theta value, sample the theta values from the beta distributions.

Take a look at the image again:

The expected rewards are $\theta = [0.6, 0.4, 0.4]$ . The greedy algorithm selects the action with the highest expected reward: that is, action 1. Thompson sampling samples from the beta distributions: in other words, $\hat{\theta_k} \sim Beta(\alpha_k, \beta_k)$ . It then pulls the arm with the maximum sampled reward.

This is genius! Why? Consider a typical sample. Sampling action 1 will give us a value very close to 0.6. Sampling action 2 will give us a value very close to 0.4. Sampling action 3, on the other hand, will give us a wide range of values, including values greater than 0.6. This means that action 2 will pretty much never be chosen, but there's a good chance of action 3 being chosen! In other words --- Thompson sampling nicely directs exploration effort towards the unknown.

I ran 10,000 simulations, and here are the results...

Bayesian inference vs frequentist inference

Once I have the results ....

I can use Fisher's exact test or the normal approximation to the binomial distribution.

By the CLT, we know that all three treatments must be normally distributed.

Bayesian inference: there is a closed form, but I can use simulation to compare the three beta distributions:

Given several beta distributions, what is the probability that one is the highest?

A paper from NIPS 2011 ("An empirical evaluation of Thompson Sampling") shows, in experiments, that Thompson Sampling beats UCB. UCB is based on choosing the lever that promises the highest reward under optimistic assumptions (i.e. the variance of your estimate of the expected reward is high, therefore you pull levers that you don't know that well). Instead, Thompson Sampling is fully Bayesian: it generates a bandit configuration (i.e. a vector of expected rewards) from a posterior distribution, and then acts as if this was the true configuration (i.e. it pulls the lever with the highest expected reward). ^[2]

Call the different SMS treatments A, B, and C, and the probability of someone installing the app $$r$$ can be conditional on either one of these treatments. The value of each treatment $Q(\cdot)$ is the expected reward given that treatment, $Q(A) = \mathbb{E}[r|A]$ .

A little wrinkle

Formulas for Bayesian A/B Testing

Why do I use Bayesian methods?

Frequentist approaches require me to set sample size for adequate power, and this sample size depends on effect size. But I have no good prior for effect size
Sample size large

Contextual bandits

Agarwal 2013 Thompson Sampling for Contextual Bandits with Linear Payoffs

Top-two Thompson Sampling for Best-Arm Identification

Simple Bayesian Algorithms for Best-Arm Identification

Now suppose we have

Here I should clarify --- suboptimal for what? For a scientific experiment where the goal is to discover a fact about the world, then we shouldn't care about exploitation at all. We want the highest possible statistical power, and that means maximising exploration. But of course in this case where money is on the line, some balance of exploration and exploitation must be struck. ↩︎
Pedro A. Ortega (https://stats.stackexchange.com/users/9572/pedro-a-ortega), Best bandit algorithm?, URL (version: 2017-03-24): https://stats.stackexchange.com/q/24027 ↩︎

A gentle introduction to the IS-PC-MR model

2019-04-21T00:00:00Z

TODO: work-in-progress.

Introduction and topics covered

Here I go through the IS-PC-MR model we learned in the first two lessons of Core Macro.

I want to be very explicit on the microfoundations: no handwaving here. For instance, I go into detail about how the PC derives from the relative steepness of the WS and PS curves.

Finally, we look at how the model works in practice.

What is the IS curve?
What is the ES, WS, MPL, PS?
What is the PC?
What is the central bank's loss function?
What is the MR and how is it derived from the central bank loss function?
a, $\alpha$ , and $\beta$
What is a supply shock?
What is a demand shock?
How the central bank responds to a supply shock
How the central bank responds to a demand shock
How the central bank can do better if it expects the shock

Assumptions of the model

We have an economy with diminishing returns. (MPL, ES)
The economy is made up purely of labourers and companies. Companies pay labourers wages, then produce goods, which labourers consume.
We have some market power in both the labour supply and labour demand markets. (WS, PS)
Changing interest rates affects consumption decisions and thus affects future output. (IS)
Output gap affects inflation due to WS/PS curves (PC)
Central bank operates with a loss function (MR)

The IS curve

IS: y*t - y_e = a(r*{t-1} - r_e)

Microfoundations

We begin with the IS curve. The IS curve says that output deviates from long-run output $y_e$ depending on the deviation of last period's interest rate.

A change in interest rates affect the output of the economy. Specifically, they affect consumption and investment decisions. (assumption 4). How does this work?

Consider this: you're a consumer, and you want to consume. Ceteris paribus, if interest rates are low (or even negative), then you would want to consume as much as you can. On the other hand, if interest is high, you'd want to reduce consumption and save more now to spend more in the future.

Higher interest rates also affect investment decisions. if you were a company wanting to build a new plant, you'd have to borrow money to do it. If the interest rate was high, it's very bohua; you'd rather keep the money.

And because $Y = (C+I+G)$ , both investment and consumption affect output.

Note the $t-1$ subscript on the interest rate: this is because central banks' interest rates only affect the economy with a lag. This is because investment decisions today have been made some time in the past, and aren't affected by current interest rates. Current interest rates only factor into deciding future investment/output.

The ES, WS, PS and MPL curves

ES: upward-sloping
WS: upward-sloping, higher than ES
MPL: downward-sloping due to diminishing returns
PS: downward-sloping, lower than MPL

The ES, WS, PS and MPL pin down the level of real wages at a certain level of output.

In a competitive labour market, the demand for labourers is given by the MPL. The supply is given by individuals' utility functions, but we shan't go into that here.

The intersection of employment supply (ES) and employment demand (MPL) is the equilibrium level of employment in a competitive market. Note that employment demand is downward sloping due to diminshing returns (see assumption 1).

However, in this model, participants have market power (assumption 3). Labour unions can sign labour contracts that give them wages above the competitive wage. This is called the wage-setting curve (WS) that lies above the ES curve.

Similarly, there is a price-setting curve (PS) that lies below the MPL curve. The intersection of these two curves is the equilibrium level of employment

Involuntary and voluntary employment

Voluntary unemployment: people who are not willing to work at the prevailing market wage Involuntary unemployment: people who are willing to work at the prevailing market wage, but cannot due to the fact that the labour unions have priced them above that.

No involuntary unemployment if workers have no market power

Effect on wages ambiguous

The effect on wages is ambiguous. It depends on the relative slopes of the WS and PS curves.

Compare these two images.

In the figure on the left, the competitive equilibrium's wage w* lies above the wage that was gotten in the noncompetitive equilibrium. In the figure on the right, the reverse is true.

The PC curve

PC: \pi_t = \pi_t^E + \alpha(y_t - y_e)

In the case of adaptive expectations,

PC: \pi*t = \pi*{t-1} + \alpha(y_t - y_e)

Microfoundations

The Phillips curve is gotten by the interaction of the WS and PS curves. The Phillips curve states that inflation in this period equals expected inflation this period, plus a shock from the output gap. But why would the output gap affect inflation? We use the WS and PS curves to explain.

With the WS and PS curves in place, we can begin to understand the Phillips curve.

The figure illustrates. Suppose we want to raise output from Y* to Y'. Well, two things will happen:

Labourers will demand higher wages to work more
Due to diminishing returns, MPL will fall: companies are willing to pay less.

Labourers will demand more to work. They'll want a real wage increase of $$\theta$$ (in red).

The companies pay that wage. But they have the last laugh---they simply set the price of their goods higher, so that they make the same profit! Specifically prices will go up such that labourers' real wages are eroded like so.

w^{'} = w^* = \theta - \gamma

(remember from assumption 2 the economy is made up completely of companies and labourers, and labourers buy the goods the companies produce)

Therefore, prices go up == inflation.

The coefficient $$\alpha$$ in the PC depends on the relative steepness of the WS and PS curves. The steeper both of them are, the larger the gap between the two curves, the larger the coefficient, and the steeper the PC will be.

The MR curve

Central bank's loss function:

L = (y - y_T)^2 + (\pi - \pi_T)^2

The MR is the central bank's best response to the Phillips curve. That is to say, it solves the optimisation problem

L = (y - y_T)^2 + (\pi - \pi_T)^2 s.t. \pi*t = \pi*{t-1} + (y_t - y_e)

So we substitute the PC into the loss function, differentiate, and rearrange to obtain

\pi = \pi^T - \frac{y_t-y_e}{\alpha \beta}

a, alpha and beta

a : Found in the IS curve. Measures how responsive the economy is to interest rate changes. - For instance, economies credit-constrained consumers will have higher a.

$\alpha$ : Found in the Phillips curve. It measures the relative steepness of the WS and PS curves. - Stronger labour unions or firms with more market power increase $\alpha$ .

$\beta$ : Found in the central bank's loss function. Measures how inflation-averse a bank is (how much output it's willing to give up in order to decrease inflation). - A more inflation-averse central bank will increase $\beta$ .

The effect of these coefficients on MR

$\beta$ : Makes MR less steep. A highly inflation-averse CB (high $\beta$ ) is going to be willing to cut a lot of output to decrease a little bit of inflation. At the limit, where $\beta \to \infty$ , the MR is a straight line: the CB will literally do anything to prevent any deviation of inflation from target.
$\alpha$ : Makes MR less steep. A greater $\alpha$ means that a small change in output gives a huge change in inflation (remember that $\alpha \propto \theta + \gamma$ . So the CB gets more "bang for your buck": it's thus very willing to adjust output in order to reduce inflation, because it can do it relatively cheaply.

But note that the effect of $\alpha$ on policy path is ambiguous

Supply shocks

TODO

Supply shocks are cost-push shocks. They affect the PC curve.

Let's examine the case of a temporary positive supply shock.

$t_0$ . The economy is at steady-state with $Y = Y^{*}, \pi = \pi^{*}, \text{and } r = r^{*}$ . A supply shock causes prices of goods to increase unexpectedly. The PC moves up from $PC_0 \to PC_1$ . Immediately, the central bank raises rates from $r^{*} \to r_1$ in order to cut output to Y_1. But as monetary policy operates with a lag, this change doesn't take effect until the next period.
$t_1$ . The PC remains elevated at $PC_1$ . But the interest rate now takes effect and causes output to fall from $Y^{*} \to Y_1$ .
$t_2 \text{ onwards}$ . The PC falls from PC_1 to PC_2. The central bank lowers rates slightly. There is a protracted adjustment of interest rates and output back to their equilibrium values.

Demand shocks

TODO

Demand shocks affect the IS curve.

Let's examine the case of a temporary positive demand shock.

$t_0$ . The economy is in eqm. A positive temporary demand shock hits, causing the IS to shift from $IS \rightarrow IS^{'}$ . Inflation rises to $\pi_1$ . The central bank immediately hikes rates to $r^{'}$ , but as monetary policy operates with a lag, this change doesn't take effect until the next period.
$t_1$ . The IS curve goes back to normal, but due to higher inflation expectations the PC shifts upwards to $PC_1$ . Due to the central bank's hiked rates, however, inflation decreases from $\pi_1 \rightarrow \pi_2$ .
$t_2 \text{ onwards}$ . There is a protracted adjustment of interest rates and output back to their equilibrium values.

How the central bank can do better if it expects the shock

TODO

If the central bank expects the shock, it can cut/hike rates one period before the shock. The moment the shock hits, output is already stabilised --- as though the shock never existed.

Explaining Acemoglu's model of directed technical change: a primer

2019-04-20T00:00:00Z

Introduction and nontechnical explanation of the paradox
We can solve the paradox by directed technical change
But why would technical change be directed? Market effects
Implications of the model, and comparisons to other models:
- Schumpeterian-Solow model

Introduction and nontechnical explanation

I first want to begin with a seeming paradox. Call this The Paradox. Consider these two facts:

The number of highly-skilled workers has increased greatly over the past few decades.
The skill premium^[1] has increased despite a huge increase in high-skilled workers.

This seems counterintuitive. We know the labour market is determined by supply and demand. Therefore, it seems like the skill premium should decrease. We must conclude that something is making the skill premium go up despite the large influx of high-skilled workers.

What could it be? The answer is directed technical change: despite the fact that there are much more high-skilled workers nowadays, high-skilled workers have become much more productive than low-skilled ones, and therefore they command a higher wage.

But why have high-skilled workers become much more productive than low-skilled ones? Here's one possibility.

The nature of technological progress this few decades (e.g. computer, microchips) has been more suited for high-skilled workers.

At first glance, this explanation seems plausible. But if you think about it, it falls apart. See Acemoglu:

At the expense of oversimplifying, we can say that the microchip could have been used to develop advanced scanners that would increase the productivity of unskilled workers, or advanced computer-assisted machines that would be used by skilled workers to replace unskilled workers. (p.31)

In other words, technology itself does not confer disproportionate benefits on either group: what matters are the innovations that build upon that technological breakthrough.

The large increase in the number of high-skilled workers has made it more profitable for innovators to develop technologies that boost their productivity.

The real reason for directed technical change is actually a market size effect. Innovators focus their efforts on developing technologies that maximise their profit, and how much they can sell their R&D for depends on how many people use their technology. The more high-skilled workers there are, the more innovators develop for them, and the larger their productivity grows. This nicely turns the supply-and-demand story on its head, and gives us our answer to the paradox.

Here, I present Acemoglu's model of directed technical change, that formalises all we've just talked about. Why Acemoglu's model specifically?

It is the only model in Oxford's macroeconomics course that talks about directed technical change/ high- vs low-skilled workers.
It nicely explains The Paradox.

Review questions

What was the empirical motivation for Acemoglu's model of directed technological change?

How can the skill premium go up even as the relative supply of high-skilled workers goes up?

Formalising the paradox

The production function, as well as our utility function, is given by

Y = [(A_H H)^\rho + (A_L L)^\rho]^{1-\rho}

There are high-skilled goods produced by high-skilled labour, and low-skilled goods produced by low-skill labour. We will ignore capital. Note here that this is a CRS production function.

We shall first disregard the technology growth function (how $A_H$ and $A_L$ change over time).

Because this is a competitive market, workers are paid their marginal productivities:

w_L \equiv MPL = A^\rho_L \left[A^\rho_L + A^\rho_H\left(\frac{H}{L}\right)^\rho\right]^\frac{1-\rho}{\rho}

w_H \equiv MPH = A^\rho_H \left[A^\rho_L \left(\frac{H}{L}\right)^{-\rho} + A^\rho_H\right]^\frac{1-\rho}{\rho}

Combining these two equations we get the skill premium $\omega$ :

\omega \equiv \frac{w_H}{w_L} = \left(\frac{A_H}{A_L}\right)^\rho \left(\frac{H}{L}\right)^{-(1-\rho)}

= \left(\frac{A_H}{A_L}\right)^{(\sigma-1)/\sigma} \left(\frac{H}{L}\right)^{-(\frac{1}{\sigma})}

where $\sigma$ is the elasticity of substitution^[2]:

\sigma \equiv \frac{1}{1-\rho}

Taking logs on both sides gives

\ln \omega = \frac{\sigma-1}{\sigma} \ln(\frac{A_h}{A_l}) - \frac{1}{\sigma}\ln(\frac{H}{L})

Let's take a look at this last equation, and plot it in a graph with x-axis $\frac{H}{L}$ . We can see that as the number of skilled workers increases ( $H \uparrow$ ), the lower the wage premium.

An increase in skilled workers ( $H^{*} \rightarrow H'$ ) leads to a decrease in wage premium (movement from $A \rightarrow B$ ).

Review questions

What is the production function stated by Acemoglu?

What is the skill premium?

What is the elasticity of substitution? How does it relate to $\rho$?

Solving the paradox by directed technical change

We have just formalised the result that as the relative ratio of high-skilled workers increases, the wage premium decreases. This is great, but how then can we explain the empirical finding that the skill premium $\omega$ has increased over the decades? The answer is a combination of both elasticity of substitution $\sigma$ and a skill-biased technical change. Looking once again at the previous equation, we see that $\omega$ can increase if the term

\frac{\sigma-1}{\sigma} \ln(\frac{A_h}{A_l})

increases.

Under what circumstances will this term increase? Well, one way is to increase $A_h$ . But an increase in $A_h$ will only increase this term if $\sigma > 1$ . If workers are not very substitutable ( $\sigma < 1$ ), an improvement in the productivity of skilled workers reduces the skill premium. This case appears paradoxical at first but is in fact quite intuitive. As Acemoglu says:

Consider, for example, a Leontieff (fixed proportions) production function. In this case, when $A_h$ increases and skilled workers become more productive, the demand for unskilled workers, who are necessary to produce more output by working with the more productive skilled workers, increases by more than the demand for skilled workers. In some sense, in this case, the increase in $A_h$ is creating an “excess supply” of skilled workers given the number of unskilled workers. This excess supply increases the unskilled wage relative to the skilled wage.

Okay. So suppose $\sigma > 1$ . Then, an increase in $A_h$ will indeed increase the wage premium by shifting the line upwards.

Figure 2 illustrates. Despite the increase in H/L, skill-biased technology change causes the wage premium to increase from $w^*$ to $w^{**}$ .

Great---we have solved the paradox by introducing skill-biased technical change. More explicitly, the relative productivity of skilled workers,

(A_h/A_l)^\frac{\sigma-1}{\sigma},

must have increased.

Acemoglu's model with endogenous directed technical change.

I've just shown that one can have an increase in the ratio of high- to low- skilled workers IF the coefficient of substitution $\sigma > 1$ AND technical change is directed towards high-skilled workers.

But that begs the questions: why was technical change directed towards high-skilled workers? Now we attempt to endogenise technical change (i.e. try and find a reason within the model why technical change was directed).

We have our utility/production function once again:

Y = [(A_H H)^\rho + (A_L L)^\rho]^{1-\rho}

Here, let's introduce some innovators. Innovators can produce innovations, $A_H$ or $A_L$ , that increase the productivity of all high- or low-skill workers. Let the fixed cost of producing an innovation be B and let the marginal cost of producing the new technology be 0. Then the marginal benefit of providing a new technology is simply the marginal increase in productivity ( $H$ or $L$ ) multiplied by the price of that good; this gives $p_H H$ and $p_L L$ respectively.

We can see that innovators will only choose to produce an innovation if they stand to make a profit: in other words only when $B < p_H H$ .

Here, there are two factors that affect innovation: price and market size.

Price effect: the larger the price of the good $p_H$ , the more profit innovators stand to make.

Market size effect: The more high- or low- skilled workers $H$ , the more profit innovators stand to make.

There is a further equilibrium condition. At equilibrium, the marginal benefit of providing both high- and low- skilled technologies must be equal. Otherwise, people would start developing more technologies of the other one to make more profit.

This gives us the following condition:

p_h H = p_l L

By consumer optimisation,

\frac{p_H}{p_L} = \frac{MU_H}{MU_L}

After all, consider the case where LHS > RHS. Then, it would be a Pareto improvement to substitute 1 unit of H for 1 unit of L.

But we know here that the production and utility functions are the same! Thus,

MU_H == MPH

and

MU_L = MPL.

Therefore, we can reuse the previous result

\frac{p_H}{p_L} = \left [ \frac{A_H H}{A_L L} \right ] ^{\rho-1}

Substituting in the equilibrium condition, and replacing $$\sigma = 1/(1- \rho)$$ gives us

\frac{A_h}{A_l} = \left(\frac{H}{L}\right)^ {\sigma - 1}

This gives the result that when $\sigma > 1$ , more technology will be produced for the higher-skilled worker if H/L increases (i.e. the market size effect dominates). In other words --- directed technical progress! This is something we previously assumed in the exogenous case, but now we have shown how directed technical progress is in fact caused by a large supply of high-skilled workers.

Okay, we've shown that if $\sigma > 1$ , when $H \uparrow$ , $\frac{A_H}{A_L} \uparrow$ . But what about the skill premium?

\omega \equiv \frac{w_H}{w_L} = \left(\frac{A_H}{A_L}\right)^\rho \left(\frac{H}{L}\right)^{-(1-\rho)}

Substituting the result we got for $\frac{A_h}{A_l}$ into the equation, as well as the fact that $\sigma \equiv \frac{1}{1-\rho}$ , we obtain

= \left(\frac{H}{L}\right)^{\sigma-2}

This result tells us that if $\sigma > 2$ , then an increase in relative supply of high-skilled workers increases the skill premium. Note that this is different from the exogenous model. In the exogenous model, as long as $$\sigma > 1$$ both the level of skill-biased technology and the skill premium increase. In the endogenous model, if $1 < \sigma < 2$ the level of technology does increase, but the skill premium actually decreases (the increase in supply outweighs the increase in technology).

Level of $\sigma$	Exogenous model	Endogenous model
$0 < \sigma < 1$	$\omega \downarrow$ $A_L \uparrow$	$\omega \downarrow$ $A_L \uparrow$
$1 < \sigma < 2$	$\omega \uparrow$ $A_H \uparrow$	$\omega \downarrow$ $A_H \uparrow$
$\sigma > 2$	$\omega \uparrow$ $A_H \uparrow$	$\omega \uparrow$ $A_H \uparrow$

Comparison to other models

In the Schumpeter-Solow case, demand for innovation is gotten from $\hat{k}$ , level of capital per effective labour unit. The greater $\hat{k}$ is, the more profit innovators stand to make.

TODO

Conclusion

TODO

The wage premium $\omega$ , is defined as $\frac{w_H}{w_L}$ . ↩︎
How do we get the elasticity of substitution?? How is it defined? TODO ↩︎

Building an automated, blockchain-connected model train diorama

2019-01-31T00:00:00Z

WIP

Introduction

Over the summer of 2018 (June-Aug 2018), I built an automated, blockchain-connected model train diorama and an accompanying visualisation. The diorama shows how changes in the locations of goods can be added in real-time to the blockchain, ensuring an immutable and unforgeable chain of provenance. This exhibit is also interactive: members of the public can scan QR codes pasted on several shipping containers, and they'll see that good's location and history on their phones update in real time.

Why a blocktrain?

My boss was part of the group promoting industry adoption of blockchain. He wanted me to find some way to i) explain how the blockchain works, and ii) show how blockchain can be used in industry.

I chose to showcase the supply chain, because this application is already being promoted by IBM and it would have more relevance to the industry experts my boss talks to.

My boss was enamoured with the idea of a moving train, having seen a model train set at the Accenture HQ. So I had the task of finding out how to use a model train set to showcase the supply chain. After several iterations, I decided on the current diorama + blockchain visualisation.

Architecture

There are four main components of this diorama:

Hyperledger Blockchain and REST API (running on a DigitalOcean instance)
Train diorama (controlled by a Raspberry Pi)
Blockchain visualisation (running on the Raspberry Pi)
Asset tracker (served by a DigitalOcean instance)

Blockchain on the cloud

I spun up a DigitalOcean instance to run the blockchain.

Hyperledger Composer comes with its own REST API.

Blockchain visualisation

The blockchain visualisation is written from scratch in HTML5 Canvas, and runs on the Raspberry Pi. It polls the Hyperledger REST API to check for updates.

I made a nice animation whereupon new blocks slide in, giving the diorama a sense of dynamism.

Asset tracker

I wanted to make the diorama interactive so I created a

People viewing the diorama can use their phones to scan the QR codes attached to several goods containers.

Building the diorama

Ideation

I bought a $20 Thomas the Train model train set just to figure out how the diorama would work. You can see the video of it here:

Starting and stopping the train

This was the hardest part of the project.

I first tried to use a weird German device called an Aufenthaltsschalter to stop and start the train automatically.

The Aufenthaltsschalter*

Jain trying to figure out the device

Complicated schematic

Finally, Eureka moment

The program running on the Raspberry Pi is very simple. It constantly listens for a voltage change on any of the reed switches, and if there is, it toggles the reed switch, which stops the train (to simulate goods unloading/loading).

It then makes a HTTP request to the Hyperledger blockchain REST API to update the locations of the goods on the train.

There were many possible approaches I considered: ultrasonic sensor, light sensor, etc

Here's a video of the train starting and stopping when toggling the reed switch.

Construction

I knew I wanted water in my diorama, as i) water looks very aesthetic, ii) I wanted to put a shipping container on the ship which people could scan, iii) Singapore is an island.

To get a nice water effect, I used this thing called "Realistic Water". We painted the styrofoam base a deep blue to get a "deep sea" effect, taped the sides to prevent water egress, then slowly poured the "Realistic Water" in, pressed in some rocks and finally left it to set overnight.

It worked quite well, I think: see for yourself.

Glamour shots

Conclusion

I greatly enjoyed this project.

It was super fun to play with model trains and fix up the diorama.

Learned a lot of new things:

Hyperledger
Reed switch, relay switch

Blocktrain

2018-08-01T00:00:00Z

Blockchain train diorama (Blocktrain)

June 2018 -- Aug 2018

Tech stack: Hyperledger Fabric/Composer, Node.js, JavaScript

After obtaining buy-in from senior management, I conceptualised and led a project that persuades industry leaders to adopt blockchain in their companies.

The project is a fully-automated blockchain demonstrator for supply chain management. It showcases how blockchain technology can be adopted in the supply chain.

To this end, I've built an automated train diorama, and three other components:

Model train diorama with IoT sensors (to showcase the system in action)
Blockchain (powered by Hyperledger)
Real-time blockchain visualisation
Asset tracker (good viewer) that works on any mobile device

Here's how it works. On the train are shipping containers with QR codes pasted on them. Scanning each QR code will pull up the asset tracker, a webpage that gives the real-time location and previous provenance of the good --- all stored immutably on the blockchain. As the train pulls up to a station, the goods are unloaded and sent to a different location (absent a robotic arm, I'm afraid one has to use his imagination for this).

When this happens, each shipping container will update its location automatically. This is stored on the blockchain. I also built a blockchain visualisation so we can see new blocks being added in real-time.

I've written a more extensive writeup about the project here, where I go into further detail.

Built with Hyperledger, Node.js, and HTML Canvas.

Phonics Game

2018-07-01T00:00:00Z

Phonics Game

July 2018

An educational game for kids aged 7-8 to learn "vowel teams", made for the Ministry of Education. I didn't know what a vowel team was until I started this project.

Built with HTML Canvas and JavaScript, with art from the (extremely) talented Shan Wei. Available to play here.

Q-Step Essay

2018-05-06T00:00:00Z

In Hilary Term 2018, we were asked to read Lijphart, learn R, and write an empirical politics essay critiquing Lijphart. The essay is

expected to provide an empirical answer to one of the three questions listed below that deal with Lijphart’s regime type classification and its consequences:

Does consensus democracy reduce social inequality?

Does consensus democracy improve economic outcomes?

Does consensus democracy improve the quality of government?

I chose the first question, "Does consensus democracy reduce social inequality?" I used an R notebook to do my scratch work and preliminary data exploration, then wrote the essay in Markdown and used pandoc to compile it.

I was pleased that this essay won the 2018 prize. Professors David Kirk and Andrew Eggers were the markers. Andy later became my thesis advisor.

The GitHub repo is available at https://github.com/lieuzhenghong/qstep-essay and the essay can be found here

Dropship Chess

2017-09-01T00:00:00Z

Dropship Chess

September 2017

A game built for the the Jack computer, a computer we build ourselves from scratch (NAND gates) as part of the (excellent) course From NAND to Tetris.

It's chess played on a 6x6 board where captured enemy pieces can be "airdropped" anywhere on the board in lieu of moving a piece, kind of like Shogi/Bughouse. White starts with two knights and Black with two bishops—so you have to capture the opponent's knights/bishops to get your own.

NAND To Tetris is an excellent overview of computer architecture. You start with a NAND gate and build full adders and mutliplexers from it. You then build basic processing and storage devices (ALU, RAM , CPU, display). Then you build an assembler, then a VM, then an interpreter, then a full-fledged parser/lexer and compiler for a high level language called Jack, and finally you write the OS for the computer in Jack! At the end of the course you've built a computer which supports a high-level language and can run a highly nontrivial program (Tetris) starting only from NAND gates. Awesome!

Dropship Chess, NAND 2 Tetris

Inspector's Gadget

2017-08-01T00:00:00Z

Inspector's Gadget

August 2017

Inspector's Gadget is a bespoke desktop application written in ElectronJS that was custom-built for a civil engineering consulting firm. It streamlines the process of writing building inspection reports. Real-world usage reports show that it decreases the time taken to write a report by up to 85%.

A building inspection report involves three things:

Photos taken of the building
Floor plan of the building
PDF report of all building defects and recommended actions taken (if any)

A building inspection report is done as follows: An engineer walks around the building and takes photos of all structural features and defects (if any). The engineer will then "tag" the floor plan---put labels on the floor plan to show where each photo was taken. Finally, the engineer will produce a PDF report which includes all the photos taken, a description of each photo, and a classification of the defect type.

Before I created this application for the firm, creating a report was a very time-consuming process. First, every photo taken was renamed in File Explorer. Then, the floor plan was tagged in Microsoft Word by manually creating Text Boxes and moving them to the desired spot. Finally, to generate the report, the engineers would paste the images one by one into the Word document. God forbid the engineer missed out one photo in the middle, as to insert the new photo all subsequent photos had to be cut-and-pasted one box down. (You can see an example in the image: the image A2-30 has been left out. That means all the images and text from A2-31 onwards have to be cut and pasted one box down to make space for the new image.)

This application streamlines every part of the process, from uploading images to quick-tagging floor plans to the final report generation. Just click an area of the floor plan to tag it. The report is generated automatically, and if you missed out a photo, that's fine---the report will reflow seamlessly.

GitHub

Building a Raspberry Pi console

2017-05-30T00:00:00Z

Yesterday, Bailin and I finished the build of the Raspberry Pi console.

This Raspberry Pi console project is the culmination of more than half a year of effort. For the last eight months, I have been teaching the kids at Ulu Pandan Stars Python (specifically Python Turtle). After much hand-holding and patient instruction the kids were able to write the code for a Pong clone (mostly) all by themselves. (We wrote the code that read from the GPIO pins).

The kids don't understand all the concepts, however, which makes me think that the project is a little too ambitious. For example, they have no idea what the ADC does, how the wiring in the Raspberry Pi works and/or how the Raspi reads the hardware inputs. I don't see this as too big a deal, though, because even if they don't understand it, at least they know about it.

Nonetheless, the kids definitely do understand a lot of things: basic programming concepts (functions, loops, conditionals), Cartesian coordinates, event-based programming, and so on. We have also been training them to talk about what they have done (the girls lack the confidence to speak in a formal setting) but there's still a lot of work that still needs to be done.

Parts used

7-inch touchscreen LCD: 118 SGD
Raspberry Pi 3: ~65 SGD
Rheostats, 2: ~2 SGD each
60mm buttons, 2: 4.50 SGD each
45mm buttons, 2: 3.50 SGD each
ADC, 3 (but we used only 1): ~7 SGD each
Box file from Popular: ~5 SGD
X-acto knife: 13 SGD
Solder iron: given to me by Snoopy

Total cost: ~250 SGD

First draft

The project started as a 'shoebox console' but we found that a shoebox was way too flimsy. Here's what that looked like:

Zhengyang with the students Video of me pressing the buttons

We used an X-acto knife to cut the holes in the box and had to solder wires onto the rheostats and the push buttons.

Improved version

Bailin was able to buy a box file from Popular for $5 so we used that instead.

Bailin soldering the buttons. His hands were shaky so I took over for the more delicate parts.

The box before we put all the stuff inside it.

Closeup of the WIP console; power bank not pictured

Video tour of the console

Side view

Closeup of the console

My mom and my cousin playing with it: the game is buggy and I will get Audrey and Gillian to fix it.

Finishing up

Today I called for an extra lesson where I got the kids to mount the final two buttons so they could claim to have done at least some of it themselves (instead of having the tutors do all the work). Here's what the final version looks like now:

Front view

3/4-view. I think it kind of looks like a face!

Closeup of the console

Actually I still don't know how the thing is wired because Zheng Yang did the lion's share of the work in wiring. I'll have to learn from Bailin how everything is wired.

This was my first time working with a Raspberry Pi as well as hardware components like rheostats and buttons: my previous projects were all software-only. I'm very grateful to Zheng Yang and Bailin for helping so much (it literally could not have been done without Zheng Yang).

The kids will finish their DSA exercise in August. Whether or not they get into DSA is out of my control but I hope that they will have the confidence and motivation to pursue similar projects in the future. STEM is the future, after all...

What's next

The kids should patch their games and do some bug-fixing.

I'd like to build a few 'demonstration games' on the console that make good use of the two rheostats (an interesting input medium).

It's my hope that this console can be used as a springboard for future programming lessons.

Building my first Telegram bot

2016-09-20T00:00:00Z

I built my first Telegram bot, Charge Bot. The first commit was 3 days ago (18th September) but I first started working on it on 27th August. Then I took a three-week break and resumed working on it on 17th September.

You can check the bot out on Telegram here. The Github repo is here.

This checked a couple of technical firsts for me:

First Telegram bot
First time I used Python to write anything substantial
First deploy on Heroku
First time using non-ASCII characters and working with UTF-8 encoding
First time using a linter (pylint)

There are also a couple of non-technical firsts:

First time building a multiplayer "game"
First time building anything my friends can use
First time building something for public consumption (apart from a blog)

It was also the first time I wrote anything for public consumption.

Overall, I am quite happy with this project. The scope was manageable and it did not balloon (probably because it was a game and, being a game, the scope is rather clearly defined; what also helped was the limitations of the Telegram medium). At the same time, I was able to incorporate user feedback and suggestions.

How it works

The Python code to write a Telegram bot is extremely trivial. It is just a while loop that sends HTTP requests with a long timeout (long polling). Upon receiving a response, it sends the next request.

There is another way which is push instead of pull (doesn't send requests: waits for Telegram to send updates to you) but it is far more involved and wasn't suitable for a first-time project.

Anyway, here's the code:

def get_updates():
    while True:
        string = URL + 'getUpdates?offset={0}&timeout={1}'.format(
            read_offset(), 3600)
        req = requests.get(string)
        handle_updates(req)

That's it. But there was a gotcha: As my background is in Javascript where all responses are asynchronous, the line req = requests.get(string) tripped me up. If you were to write that in Javascript you would get req == undefined or something like that and thus handle_updates(req) would throw an error.

For Python, the synchronous nature of the code means that the assignment operator will actually stop the evaluation until it returns a proper result. That means no callback is needed and the code becomes very simple.

Snafus

Modularisation

This was something that I had a lot of trouble with, despite Mark's best efforts to help me.

My code is divvied up as follows:

* index.py (should be main.py): is the main file, imports everything
* network.py: sends requests and handles responses
* lobby.py: code that allows players to create and join a game
* game.py: game code
* charge_keyboard.py: the inline keyboard needed for the game

I was unable to understand the idea of a function that called another function (also known as... a callback function LOL). Here's what I had trouble with:

# network.py
def setPushDataTo(func):
    global callback
    callback = func

def pushDataToDispatcher(data):
    global callback
    callback(data)

# index.py

import network

def init():
    """Initialises the app and sets up all callbacks"""
    network.setPushDataTo(dispatcher)

def dispatcher(data):
    ...

Basically, this pattern allows a function in one file to call a function in another file, without importing. The reason why we cannot import is because we don't want a two-way import (index imports network, not the other way round).

But we still need some way for the "child" to communicate with the "parent" and the way to do it is by setting the callback function in the "child" to the parent's function.

I am not sure I am explaining it clearly...I'll come back in a few days to this post and see.

Deploying on Heroku

Heroku was a real pain to set up...the documentation was lacking and I spent two nights' worth just trying to figure it out.

The gist of it is that you have to have a requirements.txt file, a Procfile and a runtime.txt file.

The "Getting Started with Heroku" tutorial mentioned the first two but not the last one!!

Basically, requirements.txt is just a list of dependencies... that's quite straightforward I still don't know what runtime.txt does but I had to put in the line python-3.5.2 in order for Heroku to deploy my app with Python 3. I needed Python 3 because of it's native Unicode handling.

What I mean by that is: when Python2 receives a response as a string, it doesn't actually use Unicode encoding. In order to get it as Unicode you have to put a u' in front of it. Python 3 does this all by itself.

The Procfile is also not straightforward. Basically the Procfile tells Heroku what kind of dyno your app is (web, worker, something else), what file to run with what program. So like this:

worker: python index.py --loglevel=info

(Weirdly, typing python3 index.py throws some weird error and I don't know why and haven't been able to Google)

Bad code

This app was developed very haphazardly, and it shows. I used tons of global variables because I'm very bad. Coming from a Javascript background I don't know how to use classes, like, at all. So it's good that I'm learning Python.

I tried refactoring but it was truly a huge pain and I actually had to rollback to the previous commit because I fucked everything up. 😭

Learning points and future plans

I want to make as many bots as I can in the future, because Telegram is by far the most-used app on my phone. It was extremely smart of the Telegram team to expose this API to developers-- this has allowed it to eclipse Whatsapp even further.

For my next app, I will definitely try and use the webhooks push method.

Heroku has a free dyno tier so I have no idea why I'm paying 8 dollars a month to DigitalOcean. I think that I will move my automated transaction tracker over to Heroku as well.

I mentioned the bad code: for my next app I am going to try and write it right and I know everyone says that but I mean it, I'm going to aim for 0 warnings on pylint!!!

Overall, this was a good learning experience, I had fun. Am pretty psyched to build my next Telegram bot!

Automated Transaction Tracker

2016-08-01T00:00:00Z

Automated Transaction Tracker

August 2016

I built the Army's first item tracker and automatic SMS reminder system (that I know of), for use in the Army's logistics branch.

It's written in React and Node.js, and uses MongoDB on the back end.

The system decreases administrative overhead by automatically keeping count of all items in the store (item tracker), and sending SMSes to remind borrowers who have borrowed items from the store but have not returned them in time. The system will also automatically notify the supervisor of the store so that appropriate disciplinary action can be taken.

This system won second prize in the Army's Annual Innovation Contest.

GitHub

Form Emailer and Form Letterer

2016-04-01T00:00:00Z

Form Emailer and Form Letterer

April 2016

A mini "software suite" that makes sending mass, personalised emails very simple. I created it because I was too lazy to send emails manually when I was an intern. It was written in Python with the help of python-docx.

Form Letterer allows you to quickly generate Word documents from a template, (e.g. documents with different names/dates/amounts), while Form Emailer lets you send personalised emails with different files attached. They synergise as you can use Form Letterer to generate unique documents and then send those documents in personalised emails with Form Emailer.

One of the "killer features" this software suite has is the ability to send unique attachments (a different attachment to each recipient), which Outlook Mail Merge lacked. Another cool feature is it supports arbitrary Python code in substitutions, which allows for amazing substitutions Mail Merge just can't support like conditional execution.

Form Letterer GitHub
Form Emailer GitHub