pub struct SemaPlusPlus<T> {
    queue_and_cv: Arc<(Mutex<VecDeque<T>>, Condvar)>,
}

impl<T> SemaPlusPlus<T> {
    pub fn send(&self, message: T) {
        let (queue_lock, cv) = &*self.queue_and_cv;
        let mut queue = queue_lock.lock().unwrap();
        let queue_was_empty = queue.is_empty();
        queue.push_back(message);
        // if a queue is not empty, then another thread must have notified and main()
        // must be awake
        if queue_was_empty {
            cv.notify_all();
        }
    }

    pub fn recv(&self) -> T {
        let (queue_lock, cv) = &*self.queue_and_cv;
        // wait_while dereferences the guard and pass the protected data to the closure,
        // when the condition is true, it releases the lock and sleep until another thread
        // notify_all
        let mut queue = cv
            .wait_while(queue_lock.lock().unwrap(), |queue| queue.is_empty())
            .unwrap();
        queue.pop_front().unwrap()
    }
}

fn main() {
    let sem: SemaPlusPlus<String> = SemaPlusPlus::new();
    let mut handlers = Vec::new();
    for i in 0..NUM_THREADS {
        let sem_clone = sem.clone();
        let handle = thread::spawn(move || {
            rand_sleep();
            sem_clone.send(format!("thread {} just finished!", i))
        });
        handlers.push(handle);
    }
    // main is the only consumer of the queue,
    // it wakes up when some thread notify_all, locks the queue, consume the message
    // and release the lock until it is notified again
    for _ in 0..NUM_THREADS {
        println!("{}", sem.recv())
    }

    for handle in handlers {
        handle.join().unwrap();
    }
}

• Mutex-based channel is inefficient:
  • it does not allow the queue producer and consumer modify the queue simultaneously.
  • It involves system calls.
• Go channels are MPMC, but it essentially implement Mutex<VecDeque<>> with fuxtex.

• Minimal lock usage.
• Channels are not the best choice for global values -> lock still required.

/// Factor the number while receiving numbers from stdin
fn main() {
    let (sender, receiver) = crossbeam_channel::unbounded();
    let mut threads = Vec::new();
    for _ in 0..num_cpus::get() {
        // The channel maintains a reference counter for senders and for receivers
        // when receiver.clone(), the receiver counter increments.
        let receiver = receiver.clone();
        threads.push(thread::spawn(move || {
            while let Ok(next_num) = receiver.recv() {
                factor_number(next_num);
            }
        }));
    }

    let stdin = std::io::stdin();
    for line in stdin.lock().lines() {
        let num = line.unwrap().parse::<u32>().unwrap();
        sender.send(num).expect("Cannot send to channel when there is no receiver");
    }

    // decrement the sender counter, when no sender, the channel is closed
    drop(sender);

    for thread in threads {
        thread.join().expect("panic");
    }
}

• Internet traffic has grown faster than hardware has increased in power.
• Two machines with commodity performance is much cheaper than one machine with 2x performance.
• Need to distribute a system’s functionality over servers using networking.
• Scale out: separate stateless computation and data storage.
• Chaos engineering: intentionally introduce failures on production, e.g., randomly terminates an entire datacenter or even a region.

• Authentication: who are you? Authorization: are you allowed?
• Prevent confidential data leakage: use a framework to handle every request, check authenticartion/authorization; use type systems.
• Elasticsearch: distributed, open source search and analytics engine for all types of data; involved in multiple data breach incidents.
  • Why: bad default settings, e.g., default username and password; accept all connections by default.
• Design principles: needs to be harder to do things wrong than to do thing right.

• Fututres are single eventual values produced by asynchronous computations.
• Keep track of in-progress operations along with associated state; each future handles one operation.

/// A simplified version of Future definition
trait Future {
type Output;
// the executor thread calls poll() to start running until it cannot progress
// cx includes a wake() function so that the future wakes up the executor when it can progress again
// when wake() is called, the executor uses cx to identify the future that can make new progress
fn poll(&mut self, cx: &mut Context) -> Poll<Self::Output>;
}

enum Poll<T> {
Ready(T),
/// Allow the single thread to work on other future
Pending,
}

• In practice, futures are composed with various combinators.

// This is just an example to show combinator,
// it is not real code, as this is hard to maintain when complexity increases
// and shared data must be propogated through the chain even if they are not always used
fn addToInbox(email_id: u64, recipient_id: u64) -> impl Future<Output = Result<(), Error>> {  // return type is tedious
    loadMessage(email_id)
        // message is needed in get_recipient, but later future needs
        .and_then(|message| get_recipient(message, recipient_id))
        .map(|(message, recipient)| recipient.verifyHasSpace(&message))
        .and_then(|(message, recipient)| recipient.addToInbox(message))
}

• Can write futures that depend on multiple concurrently-executing futures.

let cook_meal_future = futures::future::join(
    vec![
        CookMeatFuture::new(),
        CookMeatFuture::new(),
        // future may return Pending after complete 2 futures,
        // the next time when the executor calls poll(), the state is restored
        CookSoupFuture::new(),
        BakeBreadFuture::new()
    ]
);

• Syntax suger: async and await.

// An async function is a function that returns future
// the compiler transform the code into Future with poll()
async fn addToInbox(email_id: u64, recipient_id: u64) -> Result<(), Error> {
    // .await waits for a future and gets its value
    let message = loadMessage(email_id).await?;
    let recipient = get_recipient(recipient_id).await?;
    recipient.verifyHasSpace(&message)?;  // this is a normal function
    recipient.addToInbox(message).await
}

• An async function does not do anything when running it, its purpose is to produce a future that does the stuff written in the function.

One should use associated types instead of generic type parameters in a trait if there should only exist one trait implementation for any type. For instance, for any given type, the Item type should be unambiguous even if type contains generic parameters.

/* standard trait
trait Iterator {
    type Item;
    fn next(&mut self) -> Option<Self::Item>;
}
*/

#[derive(Default)]
struct MyVec<T> {  // we cannot directly implement Iterator for MyVec<T> as it does not store internal states.
    vec: Vec<T>
}

struct MyVecIter<'a, T> {
    vec: &'a Vec<T>,
    position: usize,
}

impl<'a, T> Iterator for MyVecIter<'a, T> {
    type Item = &'a T;

    fn next(&mut self) -> Option<Self::Item> {
        if self.position < self.vec.len() {
            let item = &self.vec[self.position];
            self.position += 1;
            Some(item)
        } else {
            None
        }
    }
}

impl<T> MyVec<T> {
    fn iter(&self) -> MyVecIter<'_, T> {  // this does not consume MyIter
        MyVecIter {
            vec: &self.vec,
            position: 0,
        }
    }
}

impl<'a, T> IntoIterator for &'a MyVec<T> {  // this consumes MyIter
    type Item = &'a T;
    type IntoIter = MyVecIter<'a, T>;

    fn into_iter(self) -> Self::IntoIter {
        self.iter()
    }
}

// usage
let my_vec = MyVec::<i32>::default();
let mut my_vec_iter = my_vec.into_iter();
my_vec_iter.next();

On the other hands, sometimes we want multiple trait implementations for the given type, e.g., convert a type to both String and i32, or when the type has different generic bounds:

impl From<i32> for MyType {}

impl From<String> for MyType {}

impl<T> MyTrait for Vec<T> where T: Display {}

impl<T> MyTrait for Vec<T> where T: Debug {}

There are two ways to pass into a function a type that implements a trait: static dispatch and dynamic dispatch:

fn static_dispatch(item: impl MyTrait) {}  // or static_dispatch<T: MyTrait>(item: T) {}

fn dynamic_dispatch(item: &dyn MyTrait) {}

Static dispatch tells the compiler to make a copy of the function for each T and replace each generic parameter with the concrete type provided, which is called monomorphization. In this way the compiler can optimize the generic code just at non-generic code. To avoid generating duplicated machine code, one can declare a non-generic inner function inside the generic function.

When calling dynamic_dispatch, the caller passes a trait object, which is the combination of a type that implements the trait and a pointer to its virtual method table (vtable) which holds all trait method implementations of this type. Since dyn is not resolved during the compile time, it is !Sized and hence it needs to be a wide pointer (&dyn) holder, e.g., &mut, Box or Arc.

Only object-safe traits can allow dynamic dispatch. An object-safe trait must satisfy:

1. All methods are object-safe:
   1. No generic parameters,
   2. No Self as return type, otherwise the memory layout cannot be determined during the compile time.
   3. No static methods, i.e., must take &self or &mut self, otherwise the vtable is not available.
2. Trait cannot be Sized bound, otherwise it’s against the nature of dynamic dispatch.

Broadly speaking, use static dispatch in a library, and dynamic dispatch in a binary (no other users):

pub struct DynamicLogger {
    writer: Box<dyn Write>
}

pub struct StaticLogger<W: Write> {
    writer: W
}

// usage
let file = File::create("dynamic_log.txt")?;
let dynamic_logger = DynamicLogger { writer: Box::new(file) };  // Users are forced to use dynamic dispatch

// Both works for static dispatch
let file = File::create("static_log.txt")?;
let logger = Logger { writer: file };

let file: Box<dyn Write> = Box::new(File::create("static_log.txt")?);
let logger = Logger { writer: file };

fn collect_to_map<T, S>(items: impl IntoIterator<Item = T>) -> HashMap<T, usize, S>
where
   HashMap<T, usize, S>: FromIterator<(T, usize)>
{
   items
       .into_iter()
       .enumerate()
       .map(|(i, item)| (item, i))
       .collect()
}

The trait above is better than explicitly writing T: Hash + Eq and S: BuildHasher + Default since they are implicitly required for HashMap to implement FromIterator.

impl Debug for AnyIterable
where
    // the reference to Self implements IntoIterator for any lifetime 'a
    for<'a> &'a Self: IntoIterator,
    // the item (associated type in Self) produced by iterating over the reference to Self implement Debug
    // We cannnot write <&'a Self>::Item because the associated type Item only exists in IntoIterator
    for<'a> <&'a Self as IntoIterator>::Item: Debug,
{
    // since fmt signature does not allow lifetime annotation,
    // we need to specify "the reference implements the trait for any lifetime"
    fn fmt(&self, f: &mut Formatter) -> Result<(), Error> {
        f.debug_list()  // formats items, e.g., [item1, item2]
            .entries(self) // iterates over self and debug-formatted each item
            .finish()
    }
}

The code above adds trait bounds for both outer type and the inner associated type. It implement Debug for any type that can be iterated over and whose elements are Debug.

#![feature(impl_trait_in_assoc_type)] // require nightly

#[derive(Default)]
struct MyVec<T> {
    vec: Vec<T>,
}

impl<T> IntoIterator for MyVec<T> {
    type Item = T;
    type IntoIter = impl Iterator<Item = Self::Item>; // existential type

    fn into_iter(self) -> Self::IntoIter {
        let mut vec = self.vec;
        let position = 0;
        std::iter::from_fn(move || {
            if position < vec.len() {
                let item = vec.remove(position); // consume vec
                Some(item)
            } else {
                None
            }
        })
    }
}

fn main() {
    let mut my_vec = MyVec::<i32>::default().into_iter();
    println!("{:?}", my_vec.next());
}

The code above uses existential types in the associated types to avoid creating new type MyVecIter, hence perform zero-cost type erasure.

Below is my personal solution for the problem, while it was accepted, there is space to improve it and I may come back when I have another idea.

use std::{cell::RefCell, collections::HashMap, rc::Rc};

type List = Option<Rc<RefCell<ListNode>>>;

struct ListNode {
    key: i32,
    value: i32,
    prev: List,
    next: List,
}

struct LRUCache {
    head: List,
    tail: List,
    data: HashMap<i32, Rc<RefCell<ListNode>>>,
    capacity: usize,
}

impl LRUCache {
    fn new(capacity: i32) -> Self {
        Self {
            head: None,
            tail: None,
            data: HashMap::new(),
            capacity: capacity as usize,
        }
    }

    fn move_to_front(&mut self, node: Rc<RefCell<ListNode>>) {
        // early return if the node is already the head
        if let Some(head) = &self.head {
            if Rc::ptr_eq(head, &node) {
                return;
            }
        }

        let prev = node.borrow().prev.clone();
        let next = node.borrow().next.clone();

        match (prev.as_ref(), next.as_ref()) {
            (None, None) => {
                // insert a new node only modifies the head,
                // the rest of the list is untouched.
            }
            (Some(prev), None) => {
                // get the tail node
                prev.borrow_mut().next = None;
                self.tail = Some(Rc::clone(prev));
            }
            (None, Some(_)) => {
                // get the head node is already handled
                return;
            }
            (Some(prev), Some(next)) => {
                // get a node in the middle of a list,
                // connect its prev and next
                prev.borrow_mut().next = Some(Rc::clone(next));
                next.borrow_mut().prev = Some(Rc::clone(prev));
            }
        }

        // update node to be the new head
        if let Some(old_head) = &self.head {
            node.borrow_mut().next = Some(Rc::clone(old_head));
            old_head.borrow_mut().prev = Some(Rc::clone(&node));
        } else {
            // insert a node to the empty list
            self.tail = Some(Rc::clone(&node));
        }
        node.borrow_mut().prev = None;
        self.head = Some(node);
    }

    fn get(&mut self, key: i32) -> i32 {
        let mut result = -1;
        if let Some(node) = self.data.get(&key) {
            result = node.borrow().value;
            self.move_to_front(Rc::clone(node));
        }
        result
    }

    fn put(&mut self, key: i32, value: i32) {
        if let Some(node) = self.data.get(&key) {
            let node = Rc::clone(node);
            node.borrow_mut().value = value;
            self.move_to_front(node);
        } else {
            let new_node = Rc::new(RefCell::new(ListNode {
                key,
                value,
                prev: None,
                next: None,
            }));
            // evict the tail node
            if self.data.len() >= self.capacity {
                if let Some(tail) = self.tail.take() {
                    // cannot simply take the prev as it breaks the list?
                    let prev = tail.borrow().prev.clone();
                    self.data.remove(&tail.borrow().key);
                    if let Some(prev) = prev {
                        prev.borrow_mut().next = None;
                        self.tail = Some(prev);
                    } else {
                        self.head = None;
                        self.tail = None;
                    }
                }
            }
            self.data.insert(key, Rc::clone(&new_node));
            self.move_to_front(new_node);
        }
    }
}

• Complexity is anything related to the structure of a software system that makes it hard to understand and modify the system.
• \(C = \sum_p(c_pt_p)\): The overall complexity is determined by the complexity of each part \(p\) weighted by the fraction of time developers working on that part.
  • Isolating complexity in a place where it will never be seen is as good as eliminating the complexity.
• Complexity is more apparent to readers than writers: if you write a piece of code that seems simple to you, but others think it is complex, then it is complex.

• Change amplification: a seemingly simple change requires code modifications in many different places.
• Cognitive load: developers have to spend more time learning the required information to complete a task, which leads to greater risk of bugs.
  • E.g., a function allocates memory and returns a pointer to the memory, but assumes the caller should free the memory.
  • Sometimes an approach that requires more lines of code is actually simpler as it reduces cognitive load.
• Unknown unknowns (the worst!): it is not obvious which pieces of code must be modified or which information must have to complete the task.
• The goal of a good design is for a system to be obvious: a developer can make a quick guess about what to do and yet to be confident that the guess is correct.

• Complexity is caused by two things: dependencies and obscurity.
• A dependency exists when a code cannot be understood and modified in isolation.
  • E.g., in network protocols both senders and receivers must conform to the protocol, changing code for the sender almost always requires corresponding changes at the receiver.
  • Dependencies are fundamental and cannot be completely eliminated, the goal is to make the dependencies remain simple and obvious (e.g., variable renaming are detected by compilers).
• Obscurity occurs when important information is not obvious, e.g., a variable is too generic or the documentation is inadequate.
  • Obscurity is often associated with non-obvious dependencies.
  • Inconsistency is also a major contributor, e.g., the same variable name is used for two different purposes.
  • The need for extensive documentation is often a red flag that the design is complex.
• Dependencies lead to change amplification and cognitive load; obscurity creates unknown unknowns and cognitive load.

• Requires an investment mindset to improve the system design rather than taking the fastest path to finish the current task.
• Proactive investment: try a couple of alternative designs and pick the cleanest one; imagine a few ways in which the system might need to be changed in the future; write documentation.
• Reactive investments: continuously make small improvements to the system design when a design problem becomes obvious rather than patching around it.
• The ideal design tends to emerge in bits and pieces, thus the best approach is to make lots of small investments on a continual basis, e.g., 10-20% of total development time on investments.

• A module interface contains two information: formal and informal.
• The formal part for a method is its signature; The formal interface for a class consists of the signatures for all public methods and variables.
• The informal part includes its high-level behavior and usage constraints; they can only be described using comments and cannot be ensured by the programming languages.
  • Informal aspects are larger and more complex than the formal aspects for most interfaces.
• While providing choice is good, interfaces should be designed to make the common case as simple as possible (c.f. \(C = \sum_p(c_pt_p)\)).

• An abstraction is a simplified view of an entity, which omits unimportant details.
  • The more unimportant details that are omitted from an abstraction, the better, otherwise the abstraction increases the cognitive load.
  • A detail can only be omitted if it is unimportant, otherwise obscurity occurs.
• In modular programming, each module provides an abstraction in form of its interface.
• The key to designing abstractions is to understand what is important.
  • E.g., how to choose storage blocks for a file is unimportant to users, but the rules for flushing data to secondary storage is important for databases, hence it must be visible in the file system interface.
  • Garbage collectors in Go and Java do not have interface at all.

• The leakage occurs when a design decision is reflected in multiple modules. thus creating dependencies between the modules.
  • Interface information is by definition has been leaked.
• Information can be leaked even if it does not appear in the interface, i.e., back-door leakage.
  • E.g., two classes read and write the same file format, then if the format changes, both classes need to be modified; such leakage is more pernicious than interface leakage as it is not obvious.
• If affected classes are relatively small and closely tied to the leaked information, they may need to be merged into a single class.
  • The bigger class is deeper as the entire computation is easier to be abstracted in the interface compared to separate sub-computations.
• One may also pull the leaked information out of all affected classes and create a new class to encapsulate the information, i.e., replace back-door leakage with interface leakage.
• One should avoid exposing internal data structures (e.g., return by reference) as such approach makes more work for callers, and make the module shallow.
  • E.g., instead of writing getParams() which returns a map of all parameters, one should have getParameter(String name) and getIntParameter(String name) to return a specific parameter and throw an exception if the name does not exist or cannot be converted.

• Temporal decomposition is a common cause of information leakage.
• It decompose the system into operations corresponding to the execution order.
  • E.g., A file-reading application is broken into 3 classes: read, modify and write, then both reading and writing steps have knowledge about the file format.
  • The solution is to combine the core mechanisms for reading and writing into a single class.
• Orders should not be reflected in the module structure unless different stages use totally different information.
• One should focus on the knowledge needed to perform each task, not the order in which tasks occur.

• Specialized design: use individual method in the text class to support each high-level features, e.g., backspace(Cursor cursor) deletes the character before the cursor; delete(Cursor cursor) deletes the character after the cursor; deleteSelection(Selection selection) deletes the selected section.
• The specialized design creates a high cognitive load for the UI developers: the implementation ends up with a large number of shallow methods so a UI developer had to learn all of them.
  • E.g., backspace provides a false abstraction as it does not hide the information about which character to delete.
• The specialized design also creates information leakage: abstractions related to the UI such as backspace key and selection, are reflected in the text class, increasing the cognitive load for the text class developers.
• General-purpose design define API only in terms of basic text features without reflecting the higher-level operations.
  • Only three methods are needed to modify a text: insert(Position position, String newText), delete(Position start, Position end) and changePosition(Position position, int numChars).
    • The new API uses a more generic Position to replace a specific user interface Cursor.
    • The delete key can be implemented as text.delete(cursor, text.ChangePosition(cursor, 1)), the backspace key can be implemented as text.delete(cursor, text.ChangePosition(cursor, -1)).
• The new design is more obvious, e.g., the UI developer knows which character to delete from the interface, and also has less code overall.
• The general-purpose methods can also be used for new feature, e.g., search and replace text.

• A pass-through method is a method that does little except invoke another method, whose signature is similar or identical to the callee function.
• Pass-through methods usually indicates there is not a clean division of responsibility between classes.
• Pass-through methods also create dependencies between classes.
• The solution is to refactor the classes, e.g., expose the lower level class directly to the higher level (b), redistribute the functionality (c) or merge them (d):

• A pass-through variable is a variable that is passed down through a long chain of methods.
• Pass-through variables add complexity as all intermediate methods must be aware of the existence and need to be modified when a new variable is used.
• The author’s solution is to introduce a context object which stores all application’s global states, e.g., configuration options and timeout value, and there is one context object per system instance.
• To avoid passing through the context variable, a reference to the context can be saved in other objects.
  • When a new object is created, the context reference is passed to the constructor.
• Contexts should be immutable to avoid thread-safety issues and may create non-obvious dependencies.

• Dispatcher: a method that uses its arguments to select a specific method to invoke and passes most of its arguments.
  • E.g., when a web server receives an HTTP request, it invokes a dispatcher to examine the URL and selects a specific method to handle the request.
• Polymorphous methods, e.g., len(string) and len(array) reduces cognitive load; they are usally in the same layer and do not invoke each other.
• Decorator: a wrapper that takes an existing object and extends its functionality.
• Decorators are often shallow and contain pass-through methods, one should consider following alternatives before using them:
  • Add the new functionality directly to the class if it is relatively general-purpose; or merge it with the specific use case if it is specialized.
  • Merge the new functionality with an existing decorator to make the existing decorator deeper.
  • Implement it as a stand-alone class independent of the base class, e.g., Window and ScrollableWindow.

• Example 1: instead of throwing an exception when a key does not exist in remove, simply return to ensure the key no longer exists.
• Example 2: instead of throwing an exception when trying to delete a file that is still open in other processes, mark the file for deletion to deny any processes open the old file later, and delete the file after all processed have closed the file.
• Example 3: instead of throwing an IndexOutOfBoundsExeception when substring(s, begin, end) takes out-of-range arguments, return the overlap substring.

• An exceptional condition is detected and handled at a low level in the system so that higher levels need not be aware of the condition.
• E.g., when a TCP packet is lost, the packet is resent within the implementation and clients are unaware of the dropped packets (they notice the hanging and can abort manually).
• Exception masking results in deeper classes and pulls complexity downward.

• Exception aggregation handles many special-purpose exceptions with a single general-purpose handler.
• Example 1: instead of catching the exception for each individual missing parameter, let the single top-level exception handler aggregate the error message with a single top-level try-catch block.
  • The top-level handler encapsulates knowledge about how to generate error responses, but knows nothing about specific errors.
  • Each service knows how to generate errors, but does not know how to send the response.
• Example 2: promote rare exceptions (e.g., corrupted files) to more common exceptions (e.g., server crashes) so that the same handler can be used.

• The correct process of writing comments will improve the system design.
• A significant amount of design information that was in the mind of the designer cannot be represented in code, e.g., the high-level description of a method, the motivation for a particular design.
• Comments are fundamental to abstractions: if users must read the code to use it, then there is no abstraction.
  • If there is no comment, the only abstraction of a method is its declaration which misses too much essential information to provide a useful abstraction by itself; e.g., whether end is inclusive.
• Good comments reduce cognitive load and unknown unknowns.

• Follow the comment conventions, e.g., Doxygen for C++, godoc for Go.
• Comments categories:
  • Interface: a comment block that precedes the declaration; describes the interface, e.g., overall behavior or abstraction, arguments, return values, side effects or exceptions and any requirements the caller must satisfy.
  • Data structure member: a comment next to the declaration of a field in a data structure, e.g., a variable.
  • Implementation comment: a comment inside the code to describe how the code work internally.
  • Cross-module comment: a comment describing dependencies that cross module boundaries.
• The interface and the data structure member comments are the most important and should be always present.
• Don’t repeat the code: if someone who has never seen the code can also write the comment by just looking at the code next to the comment, then the comment has no value.
• Don’t use the same words in the comment that appear in the name of the entity being described, pick words that provide additional information.
• Comments should augment the code by providing information at a different level of detail.
  • Lower-level: add precision by clarifying the exact meaning of the code.
  • Higher-level: offer intuition behind the code.

• Vague name examples: “count”, “status”, “x”, “result” in a no return method.
• It’s fine to use generic “i/j” in a loop as long as the loop does not span large.
• A name may also become too specific, e.g., delete(Range Selection) suggests the argument must be a selection, but it can also be passed with unselected range.
• If you find it difficult to come up with a name that is precise, intuitive and not too long, then it is a red flag that the variable may not have a clear design.
  • Consider alternative factorings, e.g., separate the representation into multiple variables.

• Always use the common name for and only for the given purpose, e.g., ch for a channel.
• Make sure the purpose is narrow enough that all variables with the same name have the same behavior.
  • E.g., a block purpose is not narrow as it can have different behaviors for physical and logical blocks.
• When you need multiple variables that refer to the same general thing, use the common name for each variable and add a distinguishing prefix, e.g., srcBlock and dstBlock.
• Always use i in outmost loops and j for nested loops.

• Secure, reliable and scalable.
• Scalability relies on the network message distribution efficiency; hence the IC network is divided into subnets.

• From top to down: execution, message routing, consensus, P2P.

• Enables nodes to synchronize the replicated state of the subnet by downloading the required state and verify its authenticity relative to the subnet’s chain key with the state sync protocol.
• Up-to-date nodes create a checkpoint regularly by writing the replicated state to disk, computing the state hash, the consensus layer tries to agree on the state by including the hash in the catch-up packages (CUPs).
  • A state recorded on the CUP implies the majority of nodes agree on the state and a majority of nodes can serve this state.

• A new transport protocol with TLS 1.3, new handshake, new packet structure, encryption, uni/bi-directional streams, etc.
• Low latency: zero-handshake latency for recently visited sites, no head-of-line (HOL) blocking.
  • HOL blocking: the delivery of subsequent packets in a data stream is delayed due to the need to process or retransmit an earlier packet.
• Encrypted transport: encrypt most of the packet header by using the connection IDs.
  • The connection IDs also tracks migrated connections securely.
  • Packet numbers are also encrypted to avoid cross-network correlation.
• Packetization: support multiple stream frames in a single packet.
  • Allow unaffected streams to continue processing even if one stream has packet loss.
  • Enable multiplexing of different data streams over a single connection.
• Stateless design: each QUIC packets contain enough information to be processed independently and has built-in mechanism to match responses to requests.

• If the receiver slows down the message consumption, the sender’s buffer fills up and the sender’s networking layer must take one of the three paths:
  • Propagate the backpressure to the application layer to slow down data production; would be a DoS attack vector.
  • Buffer messages (indefinitely); also an attack vector.
  • Drop egress messages; risks the liveness, i.e., no delivery guarantee.

• When a subnet node creates an artifact or receives it from its peer, it gossips this artifact to all its peers, i.e., other connected subnet nodes.
  • Each artifact eventually propagates through the whole subnet.

• Network messages to be broadcast in the subnet, e.g.,
  • Users input to canister smart contracts;
  • Protocol-originating messages, e.g., blocks produced by the consensus layer or state synchronization certification.

• P2P layer should provide the above guarantee under the following constraints:
  • Bounded-time delivery or eventual (under weaker assumptions) delivery.
  • Tolerate up to a certain threshold of dropped artifacts or byzantine nodes (1/3).
  • Each peer has its own share of the resources available at other peers and the resource usage by each peer is bounded.
  • Should be able to prioritize different artifacts.
  • Provide high throughput (rather than low latency) and avoid data duplication to utilize the network.
  • Resilient to DoS/Spam and enable encryption, authenticity and integrity.

• Simply flooding all artifacts consumes unnecessary network bandwidth, instead artifacts previews called adverts are sent first.
• An advert includes fields used by the gossip protocol and its application components (which process the messages) for integrity verification and decision-making (e.g., which artifacts to prioritize)
• Other nodes may request the corresponding artifact from one or more of its peers who send the adverts.
• In the new P2P layer, if the artifact is small enough (< 1KB), adverts are not used.

• A data structure maintained by the gossip protocol.
• Contain all available artifacts for each application component.
• P2P informs consensus and other client components about the pool changes by calling on_state_change(), each component determines its next action, e.g., validation.
  • Each call returns a set of ChangeActions corresponding to the addition and deletion of artifacts from the validated pool of a client; the corresponding adverts are then broadcasted.
• The artifacts can be persistent to non-volatile storage.
• Separated into validated and unvalidated sections; the size of each unvalidated section for each peer is bounded to prevent bad peers from filling up the pool.

• Advert queue: a priority queue of all adverts the peer has received from another peer, ordered by their priority.
• Requested set: contains all adverts whose artifacts have been requested.
• Received check cache: used to prevent duplicated artifact requests.

• Create a new advert for a new artifact added by application component locally and sends to all peers;
• Process a new advert.
• Process a new artifact.
• Handle recovery and reconnection.

• NNS manages the subnet membership, each node only requests and accepts connections with nodes in the same subnet to prevent DoS attack.
• NNS canisters guarantee that all communication between two nodes are encrypted and authenticated by TLS.

• Transport keeps buffering ongoing messages in TX queues;
• When the connection works again, transmit all buffered messages and empty TX queues.

• Transport notifies the receiver gossip protocol, sends retransmission request with artifact filter to tell the sender the latest advert the receiver has seen;
  • The receiver may not need to catch up all artifacts since they may have received the same adverts from other peers before sending the retransmission.
• When receiving a retransmission request, the sender sends all relevant adverts according to the filter through the TX queue.
• If the TX queue becomes full again, another retransmission takes place.

• P2P layer is designed to distribute small messages, which is not the case for the state sync protocol.
• To simplify the P2P layer, it is separated into 2: one for state sync and the other for the rest clients.
• The P2P layer uses a new transport component to support two async APIs: push(message, peer_id) and rpc(message, peer_id) -> response.
  • P2P periodically calls push() with the current states to advertise own current state to all peers.
  • When noticing itself is behind, it calls rpc() to request specific chunks
• Using a single TCP steam is impossible to relate requests to responses without tracking states, therefore QUIC protocol is used to multiplex multiple streams in one connection.
  • P2P layer can be completely asynchronous to better utilize CPU and bandwidth resources, e.g., congestion on state synchronization does not necessarily affect other adverts.
  • Every response is tied to a corresponding request without having to maintain states.
  • Can help dynamically prioritize traffic of different clients.

• Bounded number of active artifacts in the validated artifact pool; the consensus protocol uses checkpoints to purge artifacts periodically, hence a maximal pool size \(C\).
• Explicit expiry of artifacts; if an artifact is purged from a pool, it is no longer disseminated to peers; if no peer of a node has an artifact, the node is guaranteed to not need that artifact even if it failed to receive it.
  • During state synchronization, nodes use artifacts to update own states, once states are updated, artifacts can be safely purged after some time, e.g., to help other nodes to synchronize states.
  • Newly joined nodes use CUP for offline state sync.

• Maintained on the sender side and inferred on the receiver side.
• The table size corresponds to the number of active artifacts in the validated pool.
• Whenever an artifact is added to the validated pool, it is added in the slot table on the sender side.
  • The sender sends out a slot update message to all peers.
  • Receivers infer the slot table state based on the arrived slot update messages.
  • Deletions are implicitly propagated by new artifact reusing the slot.
  • Allows nodes to notice when an artifact no longer exists in any peer’s slot tables and can remove it from the unvalidated pool.
• Each slot maintains a version number for the slot artifact; receivers only accept update messages with higher version numbers than the one it already has.
• A lightweight thread is spawn for each slot per peer to reliably push the slot update message.
• The approach combines buffering messages and dropping messages in handling the backpressure to achieve resilience and liveness.
• Bounds on the unvalidated pool: \(C\) artifacts from an honest peer and \(2C\) from a malicious peer.

On a local machine, it’s pretty straightforward. You just grab a scraper library like go-rod and figure out the right API calls. After you’ve snagged the page, you can use an HTML parser like goquery to pull out all the menus.

The only snag with go-rod is it’s too bulky for a Render free account. It needs to install Chromium first, but Render’s 512M RAM can’t handle that. I didn’t want to hunt for another free host, so go-rod had to go.

Claude then pitched the idea of online scraping services. Most I found were expensive, aimed at heavy-duty scraping. But I lucked out with AbstractAPI, offering 1000 total requests for free. If I’m smart, that could last me about half a year. ScraperAPI seemed promising with 1000 monthly free requests. But it choked on Javascript rendering for my targeted pages even with render=true.

AbstractAPI has its quirks too. The scraped result comes out messy, full of \n and \&#34. So I had to clean it up before goquery can make sense of it.

• Define a link node:

  struct Node {
    // member initializer list, e.g., next_(nullptr) equals to next_ = nullptr
    Node(int val) : next_(nullptr), prev_(nullptr), value_(val) {}
  
    Node *next_;
    Node *prev_;
    int value_;
  };
  

• Define the iterator for the DLL:

  class DLLIterator {
  public:
    // takes in a node to mark the start of the iteration
    DLLIterator(Node *head) : curr_(head) {}
  
    // prefix increment operator (++iter)
    DLLIterator &operator++() {
      // must use -> to access the member of a pointer!
      // use . if accessing the object itself
      curr_ = curr_->next_;
      return *this;
    }
  
    // postfix increment operator (iter++)
    // the (int) is a dummy parameter to differentiate
    // the prefix and postfix increment
    DLLIterator operator++(int) {
      DLLIterator temp = *this;
      // this is a pointer to the current object
      // *this returns the iterator object
      // ++*this calls the prefix increment operator,
      // which equals to `this->operator++()`
      ++*this;
      return temp;
    }
  
    // implement the equality operator
    // an lvalue reference argument avoids the copy
    // the const in the parameter means this function
    // cannot modify the argument
    // the `const` outside the parameter list means
    // the function cannot modify `this`
    bool operator==(const DLLIterator &str) const {
      return itr.curr_ == this->curr_;
    }
  
    // implement the dereference operator to return the value
    // at the current iterator position
    int operator*() { return curr_->value_; }
  
  private:
    Node *curr_;
  };
  

• Define DLL:

  class DLL {
  public:
    DLL() : head_(nullptr), size_(0) {}
  
    // the destructor deletes nodes one by one
    ~DLL() {
      Node *current = head_;
      while (current != nullptr) {
        Node *next = current->next_;
        delete current;
        current = next;
      }
      head_ = nullptr;
    }
  
    // after the insertion `new_node` becomes the new head
    // `head` is just a pointer to the node
    void InsertAtHead(int val) {
      Node *new_node = new Node(val);
      // new_node->next points to the object pointed by head_
      new_node->next_ = head_;
  
      if (head_ != nullptr) {
        head_->prev_ = new_node;
      }
  
      head_ = new_node;
      size_ += 1;
    }
  
    DLLIterator Begin() { return DLLIterator(head_); }
  
    // returns the pointer pointing one after the last element
    // used in the loop to determine whether the iteration ends
    // e.g., `for (DLLIterator iter = dll.Begin(); iter != dll.End(); ++iter)`
    DLLIterator End() { return DLLIterator(nullptr); }
  
    Node *head_{nullptr};  // in-class initializers
    size_t size_;
  };
  

#include <iostream>
#include <memory>  // to use std::unique_ptr
#include <utility> // to use std::move

// class Point is the same as before

// takes in a unique point reference and changes its x value
void SetXTo10(std::unique_ptr<Point> &ptr) { ptr->SetX(10); }
int main() {
  // initialize an empty unique pointer
  std::unique_ptr<Point> u1;
  // initialize a unique pointer with constructors
  std::unique_ptr<Point> u2 = std::make_unique<Point>();
  std::unique_ptr<Point> u3 = std::make_unique<Point>(2, 3);
  // can treat a unique pointer as a boolean to determine whether
  // the pointer contains data
  std::cout << "Pointer u1 is " << (u1 ? "not empty" : "empty") << std::endl;
  if (u2) {
    std::cout << u2->GetX() << std::endl;
  }
  // unique_ptr has no copy constructor!
  std::unique_ptr<Point> u4 = u3; // won't compile!
  // can transfer ownership with std::move
  std::unique_ptr<Point> u5 = std::move(u3); // then u3 becomes empty!
  // pass the pointer as a reference so the ownership does not change
  SetXTo10(u5); // can still access u5 afterwards
  return 0;
}

• Note that the compiler does not prevent 2 unique pointers from pointing to the same object.

// the following code can compile
// but it causes a double deletion!
MyClass *obj = new MyClass();
std::unique_ptr<MyClass> ptr1(obj);
std::unique_ptr<MyClass> ptr2(obj);

• A mutex wrapper that provides a RAII-style of obtaining and releasing the lock.
• When the object is constructed, the locks are acquired; when the object is destructured, the locks are released.
• Better than std::mutex since it provides exception safety.

#include <iostream>
#include <mutex>

int count = 0;
std::mutex m;
void add_count() {
  // the scoped_lock constructor allows for the thread
  // to acquire the mutex m
  std::scoped_lock slk(m);
  count += 1;
  // once the function finishes, slk is destructurd and
  // m is released
}

• Allow threads to wait until a particular condition before acquiring a mutex.
• Allow other threads to signal waiting threads to alert them that the condition may be true. notify_one

#include <condition_variable>
#include <iostream>
#include <mutex>
#include <thread>

int count = 0;
std::mutex m;
// declare a condition variable
std::condition_variable cv;
void add_count_and_notify() {
  std::scoped_lock slk(m);
  count += 1;
  if (count == 2) {
    // notidy one waiting thread
    // otherwise the waiter thread hangs forever
    cv.notify_one();
  }
}
void waiter_thread() {
  // a unique_lock is movable but not copiable
  // more flexible than scoped_lock as it can unlock
  // and relock manually, while scoped_lock can only be
  // unlocked automatically.
  // scoped_lock cannot be used with condition variables
  std::unique_lock lk(m);
  cv.wait(lk, [] { return count == 2; });
  std::cout << count << std::endl;
}
int main() {
  std::thread t1(waiter_thread);
  // make t1 really waits
  std::this_thread::sleep_for(std::chrono::milliseconds(100));
  std::thread t2(add_count_and_notify);
  std::thread t3(add_count_and_notify);
  t1.join();
  t2.join();
  t3.join();
  return 0;
}

• A reader-writer lock allows multiple threads to have shared read access (no writers are allowed during read operations) and exclusive write access, i.e., no other readers or writers are allowed during the write.
• C++ does not have a specific reader-writer’s lock library, but can be emulated with std::shared_mutex, std::shared_lock and std::unique_lock.
• std::shared_mutex is a mutex that allows for both shared read-only locking and exclusive write-only locking.
• std::shared_lock can be used as an RAII-style read lock.

• std::unique_lock can be used a RAII-style exclusive write lock.

  #include <iostream>
  #include <mutex>
  #include <shared_mutex>
  #include <thread>
  
  int count = 0; // resource
  std::shared_mutex m;
  void read_value() {
    // use shared_lock to gain read-only, shared access
    std::shared_lock lk(m);
    std::cout << "Reading " + std::to_string(count) << std::endl;
  }
  
  void write_value() {
    // use unique_lock to gain exclusive access
    std::unique_lock lk(m);
    count += 3;
  }
  int main() {
    // since all 3 threads run in parallel,
    // the output is not deterministic
    std::thread t1(read_value);
    std::thread t2(write_value);
    std::thread t3(read_value);
    t1.join();
    t2.join();
    t3.join();
    return 0;
  }
  

A sticker tag bot needs two main functionalities:

• Handle direct messages to add new sticker tags.
• Handle inline queries to search for stickers using a given tag.

To implement the first functionality, I created a simple state machine with two states:

1. The initial state waits for a sticker input and then moves to the tag state.
2. The tag state waits for any text input to use as the tag for the previous sticker.

To implement the second functionality, one needs to use the InlineQueryResultCachedSticker method.

For local testing, one can use a lightweight local key-value storage to store and search sticker tags. I used BadgerDB(Golang) for example.

I noticed that the generated file ID for the same sticker is different each time, making it hard to check for duplicates when adding new tags. To address this, I added a /delete method to remove tags when needed.

I couldn’t find an official way to make a bot visible only to me. Suggested by Claude, I predefined a list of authorized users. Then I performed a sanity check before handling any messages.

First, let’s install some build dependencies:

sudo apt update
sudo apt upgrade  # update packages
sudo apt install build-essential libjansson4 libjansson-dev \
    libgnutls28-dev libtree-sitter-dev libsqlite3-dev
sudo apt install texinfo  # to generate documentation

build-essnetial should install necessary tools to build C/C++ programs such as gcc, g++, make, gdb and dpkg. The rest packages install pre-compiled libraries.

Besides these packages, there are two important packages to support Lisp native compilation:

sudo apt install ibgccjit0 libgccjit-11-dev  # 11 is my gcc version

After installing them, make sure to export the path in the current session, otherwise the compiler will not realize it.

export LD_LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/11:$LD_LIBRARY_PATH

The last thing to do is install a bunch of X and GTK-3 development libraries for Emacs GUI, and another bunch of image processing libraries.

sudo apt install libx11-dev libtiff-dev libgtk-3-dev libncurses-dev
sudo apt install libtiff5-dev libgif-dev libjpeg-dev libpng-dev libxpm-dev

Without the above packages, one may encounter the following error when configuring the Emacs build:

You seem to be running X, but no X development libraries were found. You should install the relevant development files for X and for the toolkit you want, such as Gtk+ or Motif. Also make sure you have development files for image handling, i.e. tiff, gif, jpeg, png and xpm.

At this moment, we can start to download Emacs source code:

wget https://ftp.gnu.org/gnu/emacs/emacs-29.4.tar.xz
tar xvf emacs-29.4.tar.xz
cd emacs-29.4

Then we can configure the build (i.e., generate Makefile) with the following command.

./configure --with-native-compilation --with-x-toolkit=gtk3 --without-pop

• --with-native-compilation: with this flag the Emacs source code is compiled to native machine code to achieve better performance.
  • Otherwise it is compiled to bytecode and then interpreted by Emacs virtual machine during runtime.
• --with-x-toolkit=gtk3: this is recommended by Claude.
• --without-pop: if we are not using Emacs as the email client, we don’t need to bother configure the protocol.

If everything goes well, one should see the following line in the output. If not, make sure libgccjit has been installed and exported.

Does Emacs have native lisp compiler? yes

Now we can finally start compiling the Emacs:

make -j$(nproc)

If some error occurs, we may want to start again, to do this:

sudo apt install autoconf automake
rm -f Makefile
./autogen.sh  # regenerate the configuration file
# then rebuild
make -j$(nproc)

Finally, install Emacs globally:

sudo make install

To confirm the Emacs indeed used the native Lisp compiler, one can evaluate inside the vanilla Emacs with M-: (M is Alt in WSL2):

(native-comp-available-p) ;; should return t

Congratulations! You have now installed the latest and fastest Emacs on WSL2.

• Data layout for efficient access.
• Concurrent access to data structures.

• For tuple indexing. While tuples are stored on pages with NSM or DSM, during the query the DBMS needs to quickly locate the page that stores specific tuples. It can achieve this with separately-stored hash tables, where each key can be a hash of a tuple id, and the value points the location.

• Insertion: when a collision occurs, linearly search the adjacent slots in a circular buffer until a open one is found.
• Lookup: search linearly from the first hashed slot until the desired entry or an empty slot is reached, or every slot has been iterated.
  • Requires to store both key and value in the slot.
• Deletion: simply deleting the entry prevents future lookups as it becomes empty; two solutions:
  • Replace the deleted entry with a dummy entry to tell future lookups to keep scanning.
  • Shift the adjacent entries which were originally shifted, i.e., those who were originally hashed to the same key.
    • Very expensive and rarely implemented in practice.
• The state-of-art linear probe hashing is Google absl::flat_hash_map.

• Maintains multiple hash tables with different hash functions to generate different hashes for the same key using different seeds.
• Insertion: check each table and choose one with a free slot; if none table has free slot, choose and evict an old entry and find it another table.
  • If a rare cycle happens, rebuild all hash tables with new seeds or with larger tables.
• \(O(1)\) lookups and deletion (also needs to store keys), but insertion is more expensive.
• Practical implementation maps a key to different slots in a single hash table.

• Maintains a linked list of buckets for each slot in the hash table; keys hashed to the same slot are inserted into the linked list.
• Lookup: hash to the key’s bucket and scan for it.
• Optimization: store bloom filter in the bucket pointer list to tell if a key exist in the linked list.

• Improve chained hashing to avoid letting chains grow forever.
• Allow multiple slot locations in the hash table to point to the same chain.

• Maintains a split pointer to keep track of next bucket to split, even if the pointed bucket is not overflowed.

• There are always only 2 hash functions: \((key\ mod\ n)\) and \((key\ mod\ 2n)\) where \(n\) is the length of buckets when the split pointer is at the index 0 (i.e., the bucket length at any time is \(n + index(sp)\)).

• Why does \(k\ mod\ 2n < n + sp\) hold?
  • A key is only mod by \(2n\) if the result of \((k\ mod\ n)\) is above the split pointer, i.e., \(0 \leq k\ mod\ n < sp)\).
  • Let \(r = k\ mod\ n\), then \(k = pn + r\) and \(0 \leq r < sp\).
  • Let \(r' = k\ mod\ 2n\), then \(k = q(2n) + r'\).
  • If \(p = 2m\), then we also have \(k = m(2n) + r = q(2n) + r'\), in this case \(0 \leq r = r' < sp\).
  • If \(p = 2m + 1\), then we have \(k = m(2n) + r + n = q(2n) + r'\), in this case \(n \leq r' = n + r < n + sp\).

1. await fetch(url): returns a Response object.
   • await waits for the method to complete.
2. await response.text(): returns the html string.
3. DOMParser().parseFromString(postHtml, "text/html"): returns a Document object, which is a complete DOM tree from the HTML string; DOMParser is a built-in browser API.
4. postDoc.getElementById("content"): returns an Element with the content id name.
5. content.querySelector(".post-title a"): fetches the first Element with the post-title class name, and returns the first Element the first <a> (anchor) tag.
   • . for classes, # for IDs (IDs should be unique within a page)

• content.querySelector(".taglist").remove(): removes the element from the DOM, i.e., it modifies content.

• const escapedQuery = query.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"): add a backslash to any special character in the query; $& is a special pattern used in Javascript’s replacement method.

• A high-level logical primitive to allow for transaction atomicity.
• Exposed to users when queries are run.
• Need to be able to rollback changes.

• A low-level protection primitive a DBMS uses in its internal data structures, e.g., hash tables.
• Only held when the operation is being made, like a mutex.
• Do not need to expose to users or used to rollback changes.

• Page table: An in-memory hash mapping page ids to frame locations in the buffer pool.
• Dirty flag: set by a thread whenever it modifies a page to indicate the pages must be written back to disk.
• Pin/Reference counter: tracks the number of threads that are currently accessing the page; the storage manager is not allowed to evict a page if its pin count is greater than 0.

• The buffer pool decides when to allocate a frame for a page.
• Global policies: decisions based on the overall workload, e.g., least recently used (LRU) or clock algorithm.
• Local policies: decisions applying to specific query, e.g., priority-based page replacement.

• Each database or page can have its own buffer pool and adopts local policies to reduce latch contention.
• To map a page to a buffer pool, the DBMS can use object IDs or page IDs as the key.
  • Record ID: a unique identifier for a row in a table (cf. Tuple layout post).
  • Object ID: a unique identifier for an object, used to reference a user-defined type.

• While the first set of pages is being processed, the DBMS can pre-fetch the second set into the buffer pool based on the dependency between pages.
  • E.g., If pages are index-organized, the sibling pages can be pre-fetched.

• Query cursors can reuse retrieved data.
• When a query comes while a previous query is being processed by scanning the table, the new query can attach its scanning cursor to the first query’s cursor.
• The DBMS keeps track of where the second query joined to make sure it also completes the scan.

• Scanned pages do not have to be stored in the buffer pool to avoid the overhead.
• Use cases: a query needs to read a large sequence of contiguous pages; temporary pages like sorting or joins.

• LRU maintains a timestamp of when each page was last accessed, and evicts the page with the oldest timestamp.
  • The timestamp can be stored in a queue for efficient sorting.
• Susceptible to sequential flooding, where the buffer pool is corrupted due to a sequential scan.
  • With the LRU policy the oldest pages are evicted, but they are more likely to be scanned soon.

• An approximation of LRU but replace the timestamp with a reference bit which is set to 1 when a page is accessed.
• Regularly sweeping all pages, if a bit is set to 1, reset to 0; if a bit is 0, evict the page.

• Tracks the last K accessed timestamps to predict the next accessed time, hence avoid the sequential flooding issue.

• Instead of using a global replacement policy, the DBMS make eviction decisions based on each query.
• Pages brought in by one query are less likely to evict pages that are important for other ongoing queries.
• The DBMS can predicts more accurately which pages should stay or be evicted once the query is complete, so that the buffer pool is less polluted with less useful pages.

• Transactions tell the buffer pool where pages are important based on the context of each page.

• Two ways to handle dirty pages in the buffer pool:
  • Fast: only drop clean pages.
  • Slow: write back dirty pages to ensure persistent change, and then evict them (if they will not be read again.).
• Can periodically walk through the page table and write back dirty pages in the background.

• Characterized by fast, repetitive, simple queries that operator on a small amount of data, e.g., a user adds an item to its Amazon cart and pay.
• Usually more writes than read.

• Characterized by complex read queries on large data.
• E.g., compute the most popular item in a period.

• OLTP + OLAP.

• Store all attributes for a single tuple contiguously in a single page, e.g., slotted pages.
• Pros: good for queries that need the entire tuple, e.g., OLTP.
• Cons: inefficient for scanning large data with a few attributes, e.g., OLAP.

• Store each attribute for all tuples contiguously in a block of data, i.e., column store.
• Pros: save I/O; better data compression; ideal for bulk single attribute queries like OLAP.
• Cons: slow for point queries due to tuple splitting, e.g., OLTP.
• 2 common ways to put back tuples:
  • Most common: use fixed-length offsets, e.g., the value in a given column belong to the same tuple as the value in another column at the same offset.
  • Less common: use embedded tuple ids, e.g., each attribute is associated with the tuple id, and the DBMS stores a mapping to jump to any attribute with the given tuple id.

• Rows are horizontally partitioned into groups of rows; each row group uses a column store.
• A PAX file has a global header containing a directory with each row group’s offset; each row group maintains its own header with content metadata.

• Block level: compress all tuples for the same table.
• Tuple level: compress each tuple (NSM only).
• Attribute level: compress one or multiple values within one tuple.
• Columnar level: compress one or multiple columns across multiple tuples (DSM only).

• Engineers often use a general purpose compression algorithm with lower compression ratio in exchange for faster compression/decompression.
• E.g., compress disk pages by padding them to a power of 2KBs and storing them in the buffer pool.
  • Why small chunk: must decompress before reading/writing the data every time, hence need o limit the compression scope.
• Does not consider the high-level data semantics, thus cannot utilize late materialization.

• The most common database compression scheme, support late materialization.
• Replace frequent value patterns with smaller codes, and use a dictionary to map codes to their original values.
• Need to support fast encoding and decoding, so hash function is impossible.
• Need to support order-preserving encodings, i.e., sorting codes in the same order as original values, to support range queries.
  • E.g., when SELECT DISTINCT with pattern-matching, the DBMS only needs to scan the encoding dictionary (but without DISTINCT it still needs to scan the whole column).

• Compress runs (consecutive instances) of the same value in a column into triplets (value, offset, length).
• Need to cluster same column values to maximize the compression.

• Use less bits to store an attribute.

• Use a special marker to indicate values that exceed the bit size and maintains a look-up table to store them.

• Only practical if the value cardinality is low.

• Record the difference between values; the base value can be stored in-line or in a separate look-up table.
• Can be combined with RLE encoding.

• Common prefixes or suffixes and their lengths are recorded to avoid duplication.
• Need to sort the data first.

• The Ethereum mainnet is divided into smaller interconnected networks called shards.
• Each shard processes and validates its own transactions parallel to others.
• Pros: increase scalability and participation.
• Cons: a single unit can be compromised; lead to centralization.

• Rather than storing each transaction data directly in the blockchain, the data is aggregated into a blob (binary object).
• Each blob performs erasure coding to dive the blob into multiple smaller pieces with redundancy.
• Encoded pieces are stored separately, the block header contain pointers to the piece locations without storing actual data.
• Transactions in a block may be distributed across multiple blobs.

• Allows one to encode blobs such that if at least half of the data in the blob is published, anyone in the network can reconstruct and re-publish the rest of the data.

• Validators randomly sample blob pieces to confirm the data can be reconstructed.g
• If a client cannot get enough pieces to verify the blob availability, or the blob fails the integrity check, or transactions within the blob are invalid or inconsistent with the blockchain state, the blob is rejected.

• A specific sharding implementation proposal.
• Require data availability sampling and proposer-builder separation.
• Can support hundreds of individual rollups.

• L1 scaling: optimizations directly to the Ethereum mainnet and core infrastructure, e.g., parallel EVM.
• L2 scaling: building secondary rollup layers, e.g., optimistic rollups and ZK rollups, to offload mainnet computation and storage.

• Bitcoin: uses UTXOs to track which inputs have been spent (no need to go through the entire chain).
• Ethereum: uses a nounce to track the number of transactions sent from an account, the nounce is included in the transaction and is incremented by 1 for every new transaction, and all transactions must be executed in order.

• Solana’s parallel smart contract runtime to process thousands of contracts in parallel.
• Solana transactions describe all states a transaction accesses to efficiently recognize transaction dependency and to schedule parallel execution without accessing full blockchain state.

• The data is lost once the power is off.
• Support fast random access with byte-addressable locations, i.e., can jump to any byte address and access the data.
• A.k.a memory, e.g., DRAM.

• The data is retained after the power is off.
• Block/Page addressable, i.e., in order to read a value at a particular offset, first need to load 4KB page into memory that holds the value.
• Perform better for sequential access, i.e., contiguous chunks.
• A.k.a disk, e.g., SSD (solid-state storage) and HDD (spinning hard drives).

• Close to CPU: faster, smaller, more expensive.

• As fast as DRAM, with the persistence of disk.
• Not in widespread production use.
• A.k.a, non-volatile memory.

• NAND flash drives that connect over an improved hardware interface to allow faster transfer.

• The architecture is like virtual memory: a large address space and a place for the OS to bring the pages from the disk.
• The OS way to achieve virtual memory is to use mmap to map the contents of a file in a process address space, and the OS is responsible for the data movement.
• If mmap hits a page fault, the process is blocked; however a DBMS should be able to still process other queries.
• A DBMS knows more about the data being processed (the OS cannot decode the file contents) and can do better than OS.
• Can still use some OS operations:
  • madvise: tell the OS when DBMS is planning on reading some page.
  • mlock: tell the OS to not swap ranges outs of disk.
  • msync: tell the OS to flush memory ranges out to disk, i.e., write.

• The storage that a device guarantees an atomic write, i.e., if the hardware page is 4KB and the DBMS tries to write 4KB to the disk, either all 4KB is written or none is.
• If the database page is larger than the hardware page, the DBMS requires extra measures to ensure the writing atomicity itself.

• The header keeps track of the number of used slots, the offset of the starting of each slot.
• When adding a tuple, the slot array grows from the beginning to the end, the tuple data grows from the end to the beginning; the page is full when they meet.
• Problems associated with this layout are:
  • Fragmentation: tuple deletions leave gaps in the pages.
  • Inefficient disk I/O: need to fetch the entire block to update a tuple; users could randomly jump to multiple different pages to update a tuple.

• Only allows creations of new pages and no overwrites.
• Stores the log records of changes to the tuples; the DBMS appends new log entries to an in-memory buffer without checking previous records -> fast writes.
• Potentially slow reads; can be optimized by bookkeeping the latest write of each tuple.

• Both page-oriented and log-structured storage rely on additional index to find a tuple since tables are inherently unsorted.
• In an index-organized storage scheme, the DBMS stores tuples as the value of an index data structure.
• E.g., In a B-tree indexed DBMS, the index (i.e., primary keys) are stored as the intermediate nodes, and the data is stored in the leaf nodes.

• If two tables are related, a DBMS can “pre-join” them so that the tables are on the same page.
• The read is faster since only one page is required to load, but the write is more expensive since a tuple needs more space (not free lunch in DB system!).

• Fixed length, usually stored using the DBMS native C/C++ types.
• E.g., INTEGER.

• Inexact, variable-precision numeric types; fast than arbitrary precision numbers.
• Could have rounding errors.
• E.g., REAL.

• Arbitrary precision data type stored in exact, variable-length binary representation (almost like a string) with additional meta-data (e.g., length, decimal position).
• E.g., DECIMAL.

• Represent data of arbitrary length, usually stored with a header to keep the track of the length and the checksum.
• Overflowed data is stored on a special overflow page referenced by the tuple, the overflow page can also contain pointers to next overflow pages.
• Some DBMS allows to store files (e.g., photos) externally, but the DBMS cannot modify them.
• E.g., BLOB.

• Usually represented as unit time, e.g., micro/milli-seconds.
• E.g., TIMESTAMP.

• 3 common approaches to represent nulls:
  • Most common: store a bitmap in a centralized header to specify which attributes are null.
  • Designate a value, e.g., INT32_MIN.
  • Not recommended: store a flag per attribute to mark a value is null; may need more bits to ensure word alignment.

• A serialization method to encode arbitrarily nested arrays of binary data.
• RLP provides a simple (e.g., no type), space-efficient and deterministic encoding.

• Used in Bitcoin to simplify proof of inclusion (PoI) of a transaction.
• If one computes the hash of an array of \(N\):
  • Construction complexity: \(O(n)\) time and space.
  • PoI complexity: \(O(n)\) time and space (needs all other items).

• Trie: a data structure that stores key-value pair in a key’s prefix tree.
• Patricia tree: compress trie by merging nodes on the same path.
• The structure the Patricia tree is independent of the item insertion order.
• The time complexity for add, query and deletion is \(O(K)\), where \(K\) is the key length.

• MPT is a hex-ary Merkle tree with an additional DB for hash lookup.
• There are 4 types of nodes:
  • Empty node: the null node the root points to when first creating the tree.
  • Leaf node: stores the real data, e.g., account balance.
  • Branch node: stores the pointers to at most 16 other nodes, e.g., they have the common prefix (nibbles) before and differ at the current nibble (4 bit,0-f).
  • Extension node: record the compressed common prefix for a branch node.
• Each pointer in the tree is the hash value of the child node; the real node data is stored in a separate DB that maps from a node hash to its data.
• If the child node is small, the parent node could also directly store the node data rather than the hash pointer.
• In practical implementation, the entire tree is typically stored in a KV DB, and each node is stored with its hash as the key.

• Rollup has a higher performance requirement for PoI.
• Separate the indexing and PoI with a sorted key-value arrays and a (binary) Merkle tree.
  • MPT: {Addr0: State0, Addr1: State1,...}.
  • Rollup: map: {Addr0: Id0, Addr1: Id1,...} + array: [(Addr0, State0), (Addr1, State1),...].
• When a client wants to query an account, it first gets the key id from the map, then get the state from the array.
• When a node wants to generate PoI, it follows the merkle path and collect hashes (more hashes than MPT).

• Stateless light nodes get a witness along with the new block, the witness is a PoI for the state change in the block.
• Light nodes download related state information, e.g., changed account from other full nodes, or from the portal network.

• In MPT, PoI for a branch node requires the hash values of all branches.
• KZG commitment reduce the proof size by adding a polynomial formula \(f(x)\) in the branch node, and each branch has a point \((x, y)\) such that \(y = f(x)\).
• In this way, the proof no longer requires hashes of other branches, the proof space complexity \(O(log_{16}N)\) (no 16 coefficient).

• A collection of JSON-RPC methods that all execution clients implement.
• Specify the interfaces between consensus and execution layers.

• A Rust-written toolkit for Ethereum application development.
• Consists of an Ethereum testing framework Forge; a framework to interact with the chain Cast; a local Ethereum node Anvil; and a Solidity REPL (Read-Eval-Print-Loop: an interactive environment) Chisel.

• A Rust-written EVM; responsible for executing transactions and contracts.

• A library to interact with the Ethereum and other EVM-base chains.

• Erigon: a Go-written Ethereum client implementation (execution layer).
• Staged sync: break the chain synchronization process into distinct stages in order to achieve better efficiency.

• A Rust implementation of an Ethereum full node; allows users to interact with the Ethereum blockchain.
• An execution layer that implements all Ethereum engine APIs.
• Modularity: every component is built as a library.
• Performance: uses Erigon staged-sync node architecture and other Rust libraries (e.g., Alloy, revm); tests and optimizes on Foundry.
• Database/Storage engine: MDBX.

• More nuanced than TPS.
• Allows for a clear understanding for the capacity and efficiency.
• Helps assessing the cost implications, e.g., DoS attacks.

• Determines the computational and storage costs for the execution.
• Key aspects: gas, gas cost (for each operation), gas price (in Wei), gas limit.

• Standardized performance tests for transaction processing and databases, e.g., how many transactions a system can process in a given period.
• Offer benchmarks for different scenarios, e.g., TPC-C for online transaction processing.

• State: the set of data for building and validating new Ethereum blocks.
• State growth: the accumulation of new account and new contract storage.

• JIT: convert bytecode to native machine code just before execution to bypass the VM’s interpretative process.
• AOT: compile the highest demand contracts and store them on disk, to avoid untrusted bytecode absuing native-code compilation.

• A paradigm/framework for designing distributed systems.
• Actor: each actor is an independent entity to receive, process and send messages; create new actors or modify its state.

• Each contract account has its own storage trie, which is usually stored in a KV database.

• Allow developers to focus on writing queries without managing database servers.
• Automatically scales up or down base on the workload.
• Pay-per-use pricing.

• Optimize how each system handle transactions and data.

• Spread the workload across multiple systems.

• Second order effects of above changes, e.g., on light clients.
• What is the best, average and worst case scenarios for each optimization.