Welcome!
Take a look if you are bored.
Book notes: A Philosophy of Software Design
This is a personal note for the book “A Philosophy of Software Design” (2018, John Ousterhout). John Ousterhout is the author of Tcl scripting language and the Raft protocol. This book works together with Stanford CS190.
1. Introduction
- The most fundamental problem in CS is problem decomposition: how to divide a complex problem into pieces that can be solved independently.
- Complexity increases inevitably over the life of the program, but simpler designs allow to build larger systems before complexity becomes overwhelming.
- Two general approaches:
- Making code more obvious, e.g., eliminating special cases, using identifiers in a consistent fashion.
- Encapsulate the complexity, i.e., divide a system into independent modules.
- Waterfall model: each phase of a project e.g., requirement, design, coding, testing, completes before the next phase starts, such that the entire system is designed once.
- Does not work for software since the problem of the initial design do not become apparent until implementation is well underway; then the developers need to patch around the problems, resulting in an explosion of complexity.
- Agile/Incremental development: in each iteration the design focuses on a small subset of the overall functionality, so that problems can be fixed when the system is small and later features benefit from previous experience.
- Developers should always think about design issues and complexity, and use complexity to guide the design.
2. Complexity
- Complexity is incremental.
- Developers must adopt a zero tolerance philosophy.
2.1. Definition
- Complexity is anything related to the structure of a software system that makes it hard to understand and modify the system.
- \(C = \sum_p(c_pt_p)\): The overall complexity is determined by the complexity of each part \(p\) weighted by the fraction of time developers working on that part.
- Isolating complexity in a place where it will never be seen is as good as eliminating the complexity.
- Complexity is more apparent to readers than writers: if you write a piece of code that seems simple to you, but others think it is complex, then it is complex.
2.2. Symptoms
- Change amplification: a seemingly simple change requires code modifications in many different places.
- Cognitive load: developers have to spend more time learning the required information to complete a task, which leads to greater risk of bugs.
- E.g., a function allocates memory and returns a pointer to the memory, but assumes the caller should free the memory.
- Sometimes an approach that requires more lines of code is actually simpler as it reduces cognitive load.
- Unknown unknowns (the worst!): it is not obvious which pieces of code must be modified or which information must have to complete the task.
- The goal of a good design is for a system to be obvious: a developer can make a quick guess about what to do and yet to be confident that the guess is correct.
2.3. Causes
- Complexity is caused by two things: dependencies and obscurity.
- A dependency exists when a code cannot be understood and modified in isolation.
- E.g., in network protocols both senders and receivers must conform to the protocol, changing code for the sender almost always requires corresponding changes at the receiver.
- Dependencies are fundamental and cannot be completely eliminated, the goal is to make the dependencies remain simple and obvious (e.g., variable renaming are detected by compilers).
- Obscurity occurs when important information is not obvious, e.g., a variable is too generic or the documentation is inadequate.
- Obscurity is often associated with non-obvious dependencies.
- Inconsistency is also a major contributor, e.g., the same variable name is used for two different purposes.
- The need for extensive documentation is often a red flag that the design is complex.
- Dependencies lead to change amplification and cognitive load; obscurity creates unknown unknowns and cognitive load.
3. Programming mindsets
- Tactical programming: the main focus is to get something working, e.g., a new feature or a bug fix.
- Tactical programming is short-sighted, e.g., each task contributes a reasonable compromise to finish the current task quickly.
- Once a code base turns to spaghetti, it is nearly impossible to fix.
3.1. Strategic programming
- Requires an investment mindset to improve the system design rather than taking the fastest path to finish the current task.
- Proactive investment: try a couple of alternative designs and pick the cleanest one; imagine a few ways in which the system might need to be changed in the future; write documentation.
- Reactive investments: continuously make small improvements to the system design when a design problem becomes obvious rather than patching around it.
- The ideal design tends to emerge in bits and pieces, thus the best approach is to make lots of small investments on a continual basis, e.g., 10-20% of total development time on investments.
4. Modular design
- The goal of modular design is to minimize the dependencies between modules.
- Each module consists of two parts: interface and implementation. The interface describes what the module does, the implementation specifies how it does.
- The interface consists of everything that a developer working on a different module must know in order to use the given module.
- The implementation consists of the code that carries out the promises made by the interface.
- The best modules are deep, i.e., those whose interfaces are much simpler than their implementations.
- In such cases the modification in the implementation is less likely to affect other modules.
- Small modules tend to be shallow, because the benefit they provide is negated by the cost of learning and using their interfaces.
- Classitis refers to a mistaken view that developers should minimize the amount of functionality in each new class.
- It may result in classes that are individually simple, but increases the overall complexity.
4.1. Interface
- A module interface contains two information: formal and informal.
- The formal part for a method is its signature; The formal interface for a class consists of the signatures for all public methods and variables.
- The informal part includes its high-level behavior and usage constraints; they can only be described using comments and cannot be ensured by the programming languages.
- Informal aspects are larger and more complex than the formal aspects for most interfaces.
- While providing choice is good, interfaces should be designed to make the common case as simple as possible (c.f. \(C = \sum_p(c_pt_p)\)).
4.2. Abstraction
- An abstraction is a simplified view of an entity, which omits unimportant details.
- The more unimportant details that are omitted from an abstraction, the better, otherwise the abstraction increases the cognitive load.
- A detail can only be omitted if it is unimportant, otherwise obscurity occurs.
- In modular programming, each module provides an abstraction in form of its interface.
- The key to designing abstractions is to understand what is important.
- E.g., how to choose storage blocks for a file is unimportant to users, but the rules for flushing data to secondary storage is important for databases, hence it must be visible in the file system interface.
- Garbage collectors in Go and Java do not have interface at all.
5. Information hiding
- Information hiding is the most important technique for achieving deep modules.
- Each module should encapsulate a few design information (e.g., data structures, low-level details) in the module implementation but not appear in its interface.
- Information hiding simplifies the module interface and makes it easier to evolve the system as a design change related a hidden information only affects that module.
- Making an item
private
is not the same as information hiding, as the information about the private items can still be exposed through public methods such asgetter
andsetter
. - If a particular information is only needed by a few of a class’s users, it can be partially hidden if it is accessed through separate methods, so that it is still invisible in the most common use cases.
- E.g., modules should provide adequate defaults and only callers that want to override the default need to be aware of the parameter.
5.1. Information leakage
- The leakage occurs when a design decision is reflected in multiple modules. thus creating dependencies between the modules.
- Interface information is by definition has been leaked.
- Information can be leaked even if it does not appear in the interface, i.e., back-door leakage.
- E.g., two classes read and write the same file format, then if the format changes, both classes need to be modified; such leakage is more pernicious than interface leakage as it is not obvious.
- If affected classes are relatively small and closely tied to the leaked information, they may need to be merged into a single class.
- The bigger class is deeper as the entire computation is easier to be abstracted in the interface compared to separate sub-computations.
- One may also pull the leaked information out of all affected classes and create a new class to encapsulate the information, i.e., replace back-door leakage with interface leakage.
- One should avoid exposing internal data structures (e.g., return by reference) as such approach makes more work for callers, and make the module shallow.
- E.g., instead of writing
getParams()
which returns a map of all parameters, one should havegetParameter(String name)
andgetIntParameter(String name)
to return a specific parameter and throw an exception if the name does not exist or cannot be converted.
- E.g., instead of writing
5.2. Temporal decomposition
- Temporal decomposition is a common cause of information leakage.
- It decompose the system into operations corresponding to the execution order.
- E.g., A file-reading application is broken into 3 classes: read, modify and write, then both reading and writing steps have knowledge about the file format.
- The solution is to combine the core mechanisms for reading and writing into a single class.
- Orders should not be reflected in the module structure unless different stages use totally different information.
- One should focus on the knowledge needed to perform each task, not the order in which tasks occur.
6. General-Purpose modules
- General-purpose modules can be used to address a broad range of problems such that it may find unanticipated uses in the future (cf. investment mindset).
- Special-purpose modules are specialized for today’s needs, and can be refactored to make it general-purpose when additional uses are required (cf. incremental software development).
- The author recommends a “somewhat general-purpose” fashion: the functionality should reflect the current needs, but the interface should be general enough to support multiple uses.
- The following questions can be asked to find the balance between general-purpose and special-purpose approach:
- What is the simplest interface that will cover all current needs?
- How many situations will a method be used?
- Is the API easy to use for the current needs (not go too far)?
6.1. Example: GUI text editor design
- Specialized design: use individual method in the text class to support each high-level features, e.g.,
backspace(Cursor cursor)
deletes the character before the cursor;delete(Cursor cursor)
deletes the character after the cursor;deleteSelection(Selection selection)
deletes the selected section. - The specialized design creates a high cognitive load for the UI developers: the implementation ends up with a large number of shallow methods so a UI developer had to learn all of them.
- E.g.,
backspace
provides a false abstraction as it does not hide the information about which character to delete.
- E.g.,
- The specialized design also creates information leakage: abstractions related to the UI such as backspace key and selection, are reflected in the text class, increasing the cognitive load for the text class developers.
- General-purpose design define API only in terms of basic text features without reflecting the higher-level operations.
- Only three methods are needed to modify a text:
insert(Position position, String newText)
,delete(Position start, Position end)
andchangePosition(Position position, int numChars)
.- The new API uses a more generic
Position
to replace a specific user interfaceCursor
. - The delete key can be implemented as
text.delete(cursor, text.ChangePosition(cursor, 1))
, the backspace key can be implemented astext.delete(cursor, text.ChangePosition(cursor, -1))
.
- The new API uses a more generic
- Only three methods are needed to modify a text:
- The new design is more obvious, e.g., the UI developer knows which character to delete from the interface, and also has less code overall.
- The general-purpose methods can also be used for new feature, e.g., search and replace text.
7. Layers of abstractions
- Software systems are composed into layers, where higher layers use the facilities provided by lower layers; each layer provides an abstraction different from the layers above or below it.
- E.g., a file in the uppermost layer is an array of bytes and is a memory cache of fixed-size disk blocks in the next lower layer.
- A system contains adjacent layers with similar abstractions is a red flag of class decomposition problem.
- The internal representations should be different from the abstractions that appear in the interface; if the interface and the implementation have similar abstractions, the class is shallow.
- It is more important for a module to have a simple interface than a simple implementation to benefit more user.
- Simple implementation example: throw an exception when don’t know how to handle the condition; define configuration parameters (developers should compute reasonable defaults automatically for configuration parameters).
- Simple interface example: make a text editor GUI character-oriented rather on line-oriented so users can insert and delete arbitrary ranges of text.
7.1. Pass-through methods
- A pass-through method is a method that does little except invoke another method, whose signature is similar or identical to the callee function.
- Pass-through methods usually indicates there is not a clean division of responsibility between classes.
- Pass-through methods also create dependencies between classes.
- The solution is to refactor the classes, e.g., expose the lower level class directly to the higher level (b), redistribute the functionality (c) or merge them (d):
7.2. Pass-through variables
- A pass-through variable is a variable that is passed down through a long chain of methods.
- Pass-through variables add complexity as all intermediate methods must be aware of the existence and need to be modified when a new variable is used.
- The author’s solution is to introduce a context object which stores all application’s global states, e.g., configuration options and timeout value, and there is one context object per system instance.
- To avoid passing through the context variable, a reference to the context can be saved in other objects.
- When a new object is created, the context reference is passed to the constructor.
- Contexts should be immutable to avoid thread-safety issues and may create non-obvious dependencies.
7.3. Acceptable interface duplication
- Dispatcher: a method that uses its arguments to select a specific method to invoke and passes most of its arguments.
- E.g., when a web server receives an HTTP request, it invokes a dispatcher to examine the URL and selects a specific method to handle the request.
- Polymorphous methods, e.g.,
len(string)
andlen(array)
reduces cognitive load; they are usally in the same layer and do not invoke each other. - Decorator: a wrapper that takes an existing object and extends its functionality.
- Decorators are often shallow and contain pass-through methods, one should consider following alternatives before using them:
- Add the new functionality directly to the class if it is relatively general-purpose; or merge it with the specific use case if it is specialized.
- Merge the new functionality with an existing decorator to make the existing decorator deeper.
- Implement it as a stand-alone class independent of the base class, e.g.,
Window
andScrollableWindow
.
8. Combine or separate functionality
- The goal is to reduce the system complexity as a whole and improve its modularity.
- Subdividing components create additional complexity, e.g. additional code.
- Developers should separate one general-purpose code from special-purpose code, each special-purpose code should go in a different module, e.g., pull the special-purpose code into higher layers.
- A general-purpose mechanism provides interfaces for special-purpose code to override.
- Each special-purpose code implements particular logic which is unaware by other code, including the general-purpose mechanism.
- Combining codes is most beneficial if they are closely related:
- They share information, e.g., HTTP request reader and parser.
- They share repeated pattern, e.g., may
goto
same cleanup code. - The combination simplifies the interface, e.g., each code implement a part of the solution.
- They are used together bi-directionally, e.g., a specific error message which is only invoked by one method.
- They overlap conceptually in that there is single higher-level category including both code.
- Each method should do one thing and do it completely.
- The length itself is rarely a good reason for splitting up methods.
- If a method is subdivided, users should be able to understand the child method independently, which typically means the child method is relatively general-purpose, otherwise conjoined methods are created.
9. Exception handling
- Exceptions refer to any uncommon condition that alters the normal control flow.
- E.g., bad arguments, an I/O operation fails, server timeout, packet loss, unprepared condition.
- It’s difficult to ensure that exception handling code really works, especially in distributed data-intensive systems.
- Classes with lots of exceptions have complex interfaces and are shallow.
- The best way is to reduce the number of places where exceptions have to be handled.
- The author proposes 4 techniques: define errors out of existence; exception handling; exception aggregation; crash.
- For errors that are not worth trying to handle, or occur infrequently, abortion is the simplest thing to do; e.g., there is nothing the application can do when an out-of-memory exception occurs.
- Same as exceptions, special cases can result in code riddled with
if
statements, they should be eliminated by designing the normal case in a way that automatically handles the special cases.
9.1. Define errors out of existence
- Example 1: instead of throwing an exception when a key does not exist in
remove
, simply return to ensure the key no longer exists. - Example 2: instead of throwing an exception when trying to delete a file that is still open in other processes, mark the file for deletion to deny any processes open the old file later, and delete the file after all processed have closed the file.
- Example 3: instead of throwing an
IndexOutOfBoundsExeception
whensubstring(s, begin, end)
takes out-of-range arguments, return the overlap substring.
9.2. Exception masking
- An exceptional condition is detected and handled at a low level in the system so that higher levels need not be aware of the condition.
- E.g., when a TCP packet is lost, the packet is resent within the implementation and clients are unaware of the dropped packets (they notice the hanging and can abort manually).
- Exception masking results in deeper classes and pulls complexity downward.
9.3. Exception aggregation
- Exception aggregation handles many special-purpose exceptions with a single general-purpose handler.
- Example 1: instead of catching the exception for each individual missing parameter, let the single top-level exception handler aggregate the error message with a single top-level try-catch block.
- The top-level handler encapsulates knowledge about how to generate error responses, but knows nothing about specific errors.
- Each service knows how to generate errors, but does not know how to send the response.
- Example 2: promote rare exceptions (e.g., corrupted files) to more common exceptions (e.g., server crashes) so that the same handler can be used.
10. Design it twice
- Rather than picking the first idea that comes to mind, try to pick several approaches that are radically different from each other.
- No need to pin down every feature of each alternative.
- Even if you are certain that there is only one reasonable approach, consider a second design anyway, no matter how bad it will be.
- It will be instructive to think about the weaknesses of the second design and contrast with other designs.
- It’s easier to identify the best approach if one can compare a few alternatives.
- Make a list of the pros and cons of each rough design, e.g.,
- Does one design have a simpler/ more general-purpose interface than another?
- Does one interface enable a more efficient implementation?
- The design-it-twice principle can be applied at many levels in a system, e.g., decompose into modules, pick an interface, design an implementation (simplicity and performance).
- No-one is good enough to get it right with their first try in large software system design.
- The process of devising and comparing multiple approaches teach one about the factors that make designs better or worse.
11. Write comments
11.1. Why write comments
- The correct process of writing comments will improve the system design.
- A significant amount of design information that was in the mind of the designer cannot be represented in code, e.g., the high-level description of a method, the motivation for a particular design.
- Comments are fundamental to abstractions: if users must read the code to use it, then there is no abstraction.
- If there is no comment, the only abstraction of a method is its declaration which misses too much essential information to provide a useful abstraction by itself; e.g., whether
end
is inclusive.
- If there is no comment, the only abstraction of a method is its declaration which misses too much essential information to provide a useful abstraction by itself; e.g., whether
- Good comments reduce cognitive load and unknown unknowns.
11.2. What are good comments
- Follow the comment conventions, e.g., Doxygen for C++, godoc for Go.
- Comments categories:
- Interface: a comment block that precedes the declaration; describes the interface, e.g., overall behavior or abstraction, arguments, return values, side effects or exceptions and any requirements the caller must satisfy.
- Data structure member: a comment next to the declaration of a field in a data structure, e.g., a variable.
- Implementation comment: a comment inside the code to describe how the code work internally.
- Cross-module comment: a comment describing dependencies that cross module boundaries.
- The interface and the data structure member comments are the most important and should be always present.
- Don’t repeat the code: if someone who has never seen the code can also write the comment by just looking at the code next to the comment, then the comment has no value.
- Don’t use the same words in the comment that appear in the name of the entity being described, pick words that provide additional information.
- Comments should augment the code by providing information at a different level of detail.
- Lower-level: add precision by clarifying the exact meaning of the code.
- Higher-level: offer intuition behind the code.
11.2.1. Lower-level comments
- Precision is most useful when commenting variable declarations.
- Missing details include: variable units; boundary conditions (inclusive/exclusive); whether a null value is permitted and what does it imply; who is responsible for a resource release; variable invariants that always true.
- Avoid vague comments, e.g., “current”, not explicitly state the keys and values in a map.
- Comments on a variable focuses on what the variable represents, not how it will be modified.
- E.g., instead of documenting when a boolean variable is toggled to true/false, document what true/false mean.
11.2.2. Higher-level comments
- Help the reader understand the overall intent and code structure, usually inside the code.
- More difficult to write than lower-level comments as one must think about the code in a different way.
- Comments can include why we need this code; what the code does on a high-level.
11.2.3. Interface comments
- Separate interface comments from implementation comments: interface comments provide information for someone to use rather than maintain the entity.
- A class interface comment documents the overall capability and limitation of a class and what each instance represents.
- A method interface comment include both higher-level and lower-level information.
- Starts with one or two sentences describing the method behavior and performance (e.g., whether concurrently).
- Must be very precise about each argument and the return value, and must mention any constraint and dependencies on the arguments.
- Must document any side effects that affect the system future behavior, e.g., modify a system data structure which can be read by other methods.
- Must document any preconditions that must be satisfied before a method is invoked.
11.2.4. Implementation comments
- The main goal is to help readers understand what the code is doing and why the code is needed (e.g., refer to a bug report), not how.
- For long methods, add a comment before each of the major blocks to provide a abstract description of what the block does.
- For complex loops, add a comment before the loop to describe what happens in each iteration at an intuitive level.
- Document the local variables if they are used over a large span of code.
11.2.5. Cross-module comments
- Real systems involve design decisions that affect multiple classes.
- The biggest challenge of documenting cross-module decisions is to find a place.
- E.g., when a new state is introduced, multiple updates are required to handle the state; an obvious place to document all required updates is inside the state enum where a new state will be added.
- When there is no obvious place, e.g., all updates depend on each other, the author recommends documenting them in a central design notes.
- The file is divided up into clearly labeled section, one for each major topic.
- Then in any code that relates to a topic, add a short comment “see ”X“ in designNotes”.
- The disadvantage of this approach is to keep is up-to-date as the system envolves.
12. Choose names
- A good name conveys information about what the underlying entity is and is not.
- Ask yourself: if someone sees this name in isolation without seeing its declaration or documentation, how closely can they guess?
- A good name has two properties: precision and consistency.
- The greater the distance between a name’s declaration and its last usage, the longer the name should be.
12.1. Precision
- Vague name examples: “count”, “status”, “x”, “result” in a no return method.
- It’s fine to use generic “i/j” in a loop as long as the loop does not span large.
- A name may also become too specific, e.g.,
delete(Range Selection)
suggests the argument must be a selection, but it can also be passed with unselected range. - If you find it difficult to come up with a name that is precise, intuitive and not too long, then it is a red flag that the variable may not have a clear design.
- Consider alternative factorings, e.g., separate the representation into multiple variables.
12.2. Consistency
- Always use the common name for and only for the given purpose, e.g.,
ch
for a channel. - Make sure the purpose is narrow enough that all variables with the same name have the same behavior.
- E.g., a
block
purpose is not narrow as it can have different behaviors for physical and logical blocks.
- E.g., a
- When you need multiple variables that refer to the same general thing, use the common name for each variable and add a distinguishing prefix, e.g.,
srcBlock
anddstBlock
. - Always use
i
in outmost loops andj
for nested loops.
Learning Dfinity P2P layer
This is a personal note for learning DFINITY Internet Computer (IC) P2P layer.
The learning resources include:
- IC wiki: How P2P works;
- Youtube: Inside the IC | P2P (July 2021);
- Medium: IC P2P layer’s secure scalability (Oct 2021);
- Medium: QUIC for state sync to simplify P2P (Sep 2023);
- Youtube: QUIC Tutorial (SIGCOMM 2020);
- Medium: Fully asynchronous IC P2P layer (Jan 2024).
1. Terminology
1.1. IC design principle
- Secure, reliable and scalable.
- Scalability relies on the network message distribution efficiency; hence the IC network is divided into subnets.
1.2. IC protocol stack
- From top to down: execution, message routing, consensus, P2P.
1.2.1. Execution layer
- Manages a safe environment for deterministic execution of software messages.
1.2.2. Message routing layer
- Routes user and system-generated messages between subnets.
- Manages the input and output queues for applications.
- Schedules messages for execution.
1.2.3. Consensus layer
- Arranges messages received from users and different subnets into blocks before delivering them to the message routing layer.
1.2.4. P2P layer
- Collects and advertises messages from users and other nodes in the same subnet, e.g., IC consensus protocol, the state sync protocol.
- Main challenges: security, performance and scalability.
1.3. State sync
- Enables nodes to synchronize the replicated state of the subnet by downloading the required state and verify its authenticity relative to the subnet’s chain key with the state sync protocol.
- Up-to-date nodes create a checkpoint regularly by writing the replicated state to disk, computing the state hash, the consensus layer tries to agree on the state by including the hash in the catch-up packages (CUPs).
- A state recorded on the CUP implies the majority of nodes agree on the state and a majority of nodes can serve this state.
1.4. QUIC protocol
- A new transport protocol with TLS 1.3, new handshake, new packet structure, encryption, uni/bi-directional streams, etc.
- Low latency: zero-handshake latency for recently visited sites, no head-of-line (HOL) blocking.
- HOL blocking: the delivery of subsequent packets in a data stream is delayed due to the need to process or retransmit an earlier packet.
- Encrypted transport: encrypt most of the packet header by using the connection IDs.
- The connection IDs also tracks migrated connections securely.
- Packet numbers are also encrypted to avoid cross-network correlation.
- Packetization: support multiple stream frames in a single packet.
- Allow unaffected streams to continue processing even if one stream has packet loss.
- Enable multiplexing of different data streams over a single connection.
- Stateless design: each QUIC packets contain enough information to be processed independently and has built-in mechanism to match responses to requests.
1.5. Backpressure in blockchain
- If the receiver slows down the message consumption, the sender’s buffer fills up and the sender’s networking layer must take one of the three paths:
- Propagate the backpressure to the application layer to slow down data production; would be a DoS attack vector.
- Buffer messages (indefinitely); also an attack vector.
- Drop egress messages; risks the liveness, i.e., no delivery guarantee.
2. P2P layer
- Send out artifacts created by the above layers.
- Receive, validate and distribute artifacts from users and other nodes.
- Guarantees the secure eventual broadcast delivery to all nodes.
- asynchronous communication network assumption provides no time upper bound.
- Provide bounded time delivery under some network assumptions.
2.1. Gossip protocol
- When a subnet node creates an artifact or receives it from its peer, it gossips this artifact to all its peers, i.e., other connected subnet nodes.
- Each artifact eventually propagates through the whole subnet.
2.2. Artifacts
- Network messages to be broadcast in the subnet, e.g.,
- Users input to canister smart contracts;
- Protocol-originating messages, e.g., blocks produced by the consensus layer or state synchronization certification.
2.3. Constraints
- P2P layer should provide the above guarantee under the following constraints:
- Bounded-time delivery or eventual (under weaker assumptions) delivery.
- Tolerate up to a certain threshold of dropped artifacts or byzantine nodes (1/3).
- Each peer has its own share of the resources available at other peers and the resource usage by each peer is bounded.
- Should be able to prioritize different artifacts.
- Provide high throughput (rather than low latency) and avoid data duplication to utilize the network.
- Resilient to DoS/Spam and enable encryption, authenticity and integrity.
2.4. Adverts
- Simply flooding all artifacts consumes unnecessary network bandwidth, instead artifacts previews called adverts are sent first.
- An advert includes fields used by the gossip protocol and its application components (which process the messages) for integrity verification and decision-making (e.g., which artifacts to prioritize)
- Other nodes may request the corresponding artifact from one or more of its peers who send the adverts.
- In the new P2P layer, if the artifact is small enough (< 1KB), adverts are not used.
2.4.1. Advert priority
- Consensus provides the gossip protocol with a priority function, which takes an advert and its attributes and returns a priority value.
- Each P2P client decide how to request the artifact (e.g.,drop or fetch immediately).
2.5. Artifact pool
- A data structure maintained by the gossip protocol.
- Contain all available artifacts for each application component.
- P2P informs consensus and other client components about the pool changes by calling
on_state_change()
, each component determines its next action, e.g., validation.- Each call returns a set of
ChangeActions
corresponding to the addition and deletion of artifacts from the validated pool of a client; the corresponding adverts are then broadcasted.
- Each call returns a set of
- The artifacts can be persistent to non-volatile storage.
- Separated into validated and unvalidated sections; the size of each unvalidated section for each peer is bounded to prevent bad peers from filling up the pool.
2.6. Peer context
- Advert queue: a priority queue of all adverts the peer has received from another peer, ordered by their priority.
- Requested set: contains all adverts whose artifacts have been requested.
- Received check cache: used to prevent duplicated artifact requests.
2.7. Gossip protocol events
- Create a new advert for a new artifact added by application component locally and sends to all peers;
- Process a new advert.
- Process a new artifact.
- Handle recovery and reconnection.
2.7.1. Process a new advert received from peer \(i\)
- If the advert is already in peer \(i\)’s received check cache, or the priority is “drop”, ignore;
- If the advert is not in the pool, adds to the advert queue for peer \(i\);
- If enough space for peer \(i\)’s unvalidated section in the artifact pool, call
download_next(i)
(with a timeout) to ask for the next artifact with the highest priority (not necessarily corresponds to the last received advert). download_next(i)
sends the request to peer \(i\) and moves the advert with the highest priority from advert queue to the requested set of peer \(i\);- If
download_next(i)
is timeout, check whether the advert is also received from other advert queue and try to fetch it from them before retrying with peer \(i\) (as peer \(i\) may be misbehaving).
2.7.2. Process a new received artifact from peer \(i\)
- First check the received artifact has been requested and verify its integrity;
- Remove all corresponding adverts from all advert queues and requested sets;
- Add the artifact to peer \(i\)’s unvalidated pool and wait for the client component to validate;
- Add the artifact hash to peer \(i\)’s received check cache for a grace period to ignore further adverts for the same artifact;
- If the artifact is removed from the unvalidated section later (e.g., the artifact is invalid), the application component may request it again from peer \(i\), with the received cache mechanism the gossip protocol does not send out the request again in a short time.
- If enough space for peer \(i\)’s unvalidated section, call
download_next(i)
.
2.8. Common attacks
2.8.1. Sybil attack
- An attack creates multiple nodes to gain influences.
2.8.2. Eclipse attack
- All peers of a correct node are byzantine nodes to disconnect the correct node from the subnet (though they cannot send spoofed artifacts due to artifact authentication).
- Mitigation: use overlays with large min cut and expansion so that at least one peer is correct for each node.
- Each node uses a different overlap.
- Small enough subnets can be a complete graph, large subnets use more sparse overlays.
2.8.3. Routing table poisoning
- Malicious nodes provide false routing information to disrupt network topology.
2.8.4. Bandwidth hogging
- Attackers consume excessive network resources.
2.9. Security
- NNS manages the subnet membership, each node only requests and accepts connections with nodes in the same subnet to prevent DoS attack.
- NNS canisters guarantee that all communication between two nodes are encrypted and authenticated by TLS.
3. Transport component
- Below the P2P component to main the actual network connections between peers.
- Have its own send buffers and internal heartbeat mechanism, which are important for bounded time delivery.
- Frame gossip message with its own 7 headers.
3.1. Transient connection disturbances
- Transport keeps buffering ongoing messages in TX queues;
- When the connection works again, transmit all buffered messages and empty TX queues.
3.2. Long disconnection with full TX queue
- Transport notifies the receiver gossip protocol, sends retransmission request with artifact filter to tell the sender the latest advert the receiver has seen;
- The receiver may not need to catch up all artifacts since they may have received the same adverts from other peers before sending the retransmission.
- When receiving a retransmission request, the sender sends all relevant adverts according to the filter through the TX queue.
- If the TX queue becomes full again, another retransmission takes place.
4. State sync protocol
- Nodes periodically advertise all locally available state versions with adverts;
- A version is the block height to which the state corresponds, and the state hash.
- If a node sees a more recent CUP, it can conclude it has fallen behind and can request the state from the peer which sends the CUP;
- The protocol ensures unchanged pieces of a state are not re-downloaded, as the state can be viewed as a file tree and each file is split into chunks.
- A node can simultaneously request chunks from multiple peers like BitTorrent.
- The resuming node starts by requesting the manifest chunk, which contains a list of all files and chunks as well as their hashes the state contains;
- The manifest is peer-agnostic and the manifest hash is included in the CUP.
- Once the manifest hash is verified, one can conclude all file and chunk hashes are authentic.
- The node then request missing chunks from multiple peers.
4.1. Monolithic P2P layer not suitable for the state sync
- P2P layer is designed to distribute small messages, which is not the case for the state sync protocol.
- To simplify the P2P layer, it is separated into 2: one for state sync and the other for the rest clients.
- The P2P layer uses a new transport component to support two async APIs:
push(message, peer_id)
andrpc(message, peer_id) -> response
.- P2P periodically calls
push()
with the current states to advertise own current state to all peers. - When noticing itself is behind, it calls
rpc()
to request specific chunks
- P2P periodically calls
- Using a single TCP steam is impossible to relate requests to responses without tracking states, therefore QUIC protocol is used to multiplex multiple streams in one connection.
- P2P layer can be completely asynchronous to better utilize CPU and bandwidth resources, e.g., congestion on state synchronization does not necessarily affect other adverts.
- Every response is tied to a corresponding request without having to maintain states.
- Can help dynamically prioritize traffic of different clients.
5. Fully QUIC-based P2P layer
- IC’s P2P layer stops using TCP altogether, which means a shift to a fully asynchronous implementation of the P2P layer.
- Each request is sent as a new QUIC stream and handled independently from other requests.
- Each client (e.g., consensus,state sync, key distribution) uses a separate instance of the P2P layer (state sync uses the specific one).
- Uses a new abstract data structure slot table to track the content of the validated artifact pool and the process of updating the peers.
- Reduces the block rate under heavy load.
- Will eventually shift all clients to use the new P2P layer.
5.1. Properties for consensus-related clients
- Bounded number of active artifacts in the validated artifact pool; the consensus protocol uses checkpoints to purge artifacts periodically, hence a maximal pool size \(C\).
- Explicit expiry of artifacts; if an artifact is purged from a pool, it is no longer disseminated to peers; if no peer of a node has an artifact, the node is guaranteed to not need that artifact even if it failed to receive it.
- During state synchronization, nodes use artifacts to update own states, once states are updated, artifacts can be safely purged after some time, e.g., to help other nodes to synchronize states.
- Newly joined nodes use CUP for offline state sync.
5.2. Slot table
- Maintained on the sender side and inferred on the receiver side.
- The table size corresponds to the number of active artifacts in the validated pool.
- Whenever an artifact is added to the validated pool, it is added in the slot table on the sender side.
- The sender sends out a slot update message to all peers.
- Receivers infer the slot table state based on the arrived slot update messages.
- Deletions are implicitly propagated by new artifact reusing the slot.
- Allows nodes to notice when an artifact no longer exists in any peer’s slot tables and can remove it from the unvalidated pool.
- Each slot maintains a version number for the slot artifact; receivers only accept update messages with higher version numbers than the one it already has.
- A lightweight thread is spawn for each slot per peer to reliably push the slot update message.
- The approach combines buffering messages and dropping messages in handling the backpressure to achieve resilience and liveness.
- Bounds on the unvalidated pool: \(C\) artifacts from an honest peer and \(2C\) from a malicious peer.
Build a free Telegram Mensa bot
1. Previously on my Telegram bot
In the post Build a free Telegram sticker tag bot, I detailed the process of integrating various online services to create a sticker tag bot. After using it for a month, I encountered an intriguing production issue: on one occasion, the inline query result was incomplete. Interestingly, restarting the Render deployment resolved the problem, though I never fully understood the root cause.
Despite this minor hiccup, the sticker tag bot has proven to be reliable most of the time. However, I found myself not utilizing it as frequently as anticipated. This was primarily because sending a recent sticker directly is often more convenient than invoking the bot by name.
At the conclusion of that post, I hinted at a second functionality I had in mind for the bot: a Mensa bot designed to inform me about the daily offerings at each Mensa (university cafeteria).
2. Why do I need a Mensa bot
One mobile app I frequently use on weekdays is Mensa, which lists daily menus for each Zürich Mensa to help people make their most important decision of the day. However, it was merely an inconvenience for myself that the app lacked images of the meals. I found it difficult to imagine the dishes based solely on the menu descriptions. To fix this, I decide to add the meal images myself.
3. How to get the data
I couldn’t find an official API for this, so I decided to scrape the webpage myself. Here’s where things got tricky: the Mensa web pages use JavaScript to render content. This meant I couldn’t just grab the page - I needed a browser to run the JavaScript first.
3.1. Scrape menus locally
3.2. Scrape menus on Render
The only snag with go-rod
is it’s too bulky for a Render free account.
It needs to install Chromium first, but Render’s 512M RAM can’t handle that.
I didn’t want to hunt for another free host, so go-rod
had to go.
Claude then pitched the idea of online scraping services.
Most I found were expensive, aimed at heavy-duty scraping.
But I lucked out with AbstractAPI, offering 1000 total requests for free.
If I’m smart, that could last me about half a year.
ScraperAPI seemed promising with 1000 monthly free requests.
But it choked on Javascript rendering for my targeted pages even with render=true
.
AbstractAPI has its quirks too.
The scraped result comes out messy, full of \n
and \"
.
So I had to clean it up before goquery
can make sense of it.
4. Performance
AbstractAPI’s free plan only lets you make one request per second. That’s kinda slow when you factor in page rendering and parsing time. The full HTML page is a whopping 3M, so I’ve gotta wait a bit for each bot request.
I thought about caching results to cut down on requests. But here’s the thing: menu images usually update right before meal times. If I didn’t catch the image last time, I need the latest scoop. So, I end up scraping fresh data for each Mensa every single time.
5. Conclusion
Here’s a quick look at what the bot can do now: it tags stickers and scrapes Mensa menus. Keep in mind the GIF is sped up to twice the normal speed.
As in the previous post, I aim to demonstrate that while Internet services can be costly, there often remain free solutions for building hobby projects. This may require more extensive research and additional processing, but it’s still feasible. I hope this continues to be the case in the years to come.
The repository is also public now.
C++ feature introduction
This is a personal not for the CMU 15-445 C++ bootcamp along with some explanation from Claude.ai.
1. Namespace
- Provides scopes to identifiers with
::
. Namespaces can be nested.
#include <iostream> namespace ABC { void spam(int a) { std::cout << "Hello from ABC::spam" << std::endl; } // declare a nested namespace namespace DEF { void bar(float a) { std::cout << "Hello from ABC::DEF::bar" << std::endl; } void use_spam(int a) { ABC::spam(a); // no difference with ABC::spam(a) if DEF // does not have a spam function spam(a); } } // namespace DEF void use_DEF_bar(float a) { // if a namespace outside of DEF wants to use DEF::bar // it must use the full namespace path ABC::DEF::bar DEF::bar(a); } } // namespace ABC
- Two namespaces can define functions with the same name and signatures.
- Name resolution rules: first check in the current scope, then enclosing scopes, finally going outward until it reaches the global scope.
- Can use
using namespace B
to use identifiers inB
in the current scope without specifyingB::
, this is not a good practice. - Can also only bring certain members of a namespace into the current scope, e.g.,
using C::eggs
.
2. Wrapper class
Used to manage a resource, e.g., memory, file sockets, network connections.
class IntPtrManager { private: // manages an int* to access the dynamic memory int *ptr_; };
- Use the RAII (Resource Acquisition is Initialization) idea: tie the lifetime of a resource to the lifetime of an object.
- Goal: ensure resources are released even if an exception occurs.
Acquisition: resources are acquired in the constructor.
class IntPtrManager { public: // the constructor initializes a resource IntPtrManager() { ptr_ = new int; // allocate the memory *ptr_ = 0; // set the default value } // the second constructor takes an initial value IntPtrManager(int val) { ptr_ = new int; *ptr_ = val; } };
Release: resources are released in the destructor.
class IntPtrManager { public: ~IntPtrManager() { // ptr_ may be null after the move // don't delete a null pointer if (ptr_) { delete ptr_; } } };
A wrapper class should not be copyable to avoid double deletion of the same resource in two destructors.
class IntPtrManager { public: // delete copy constructor IntPtrManager(const IntPtrManager &) = delete; // delete copy assignment operator IntPtrManager &operator=(const IntPtrManager &) = delete; };
A wrapper class is still moveable from different lvalues/owners.
class IntPtrManager { public: // a move constructor // called by IntPtrManager b(std::move(a)) IntPtrManager(IntPtrManager &&other) { // while other is a rvalue reference, other.ptr_ is a lvalue // therefore a copy happens here, not a move // in the constructor this.ptr_ has not pointed to anything // so no need to delete ptr_ ptr = other.ptr_; other.ptr_ = nullptr; // other.ptr_ becomes invalud } // move assignment operator // this function is used by c = std::move(b) operation // note that calling std::move() does not require implementing this operator IntPtrManager &operator=(IntPtrManager &&other) { // a self assignment should not delete its ptr_ if (ptr_ == other.ptr_) { return *this; } if (ptr_) { delete ptr_; // release old resource to avoid leak } ptr_ = other.ptr_; other.ptr_ = nullptr; // invalidate other.ptr_ return *this; } };
3. Iterator
Iterators, e.g., pointers, are objects that point to an element inside a container.
int *array = malloc(sizeof(int) * 10); int *iter = array; int zero_elem = *iter; // use ++ to iterate through the C style array iter++; // deference the operator to return the value at the iterator int first_elem = *iter;
- Two main components of an iterator:
- Dereference operator
*
: return the value of the element of the current iterator position. - Increment
++
: increment the iterator’s position by 1- Postfix
iter++
: return the iterator before the increment (Iterator
). - Prefix
++iter
: return the result of the increment (Iterator&
). ++iter
is more efficient.
- Postfix
- Dereference operator
- Often used to access and modify elements in C++ STL containers.
3.1. A doubly linked list (DLL) iterator
Define a link node:
struct Node { // member initializer list, e.g., next_(nullptr) equals to next_ = nullptr Node(int val) : next_(nullptr), prev_(nullptr), value_(val) {} Node *next_; Node *prev_; int value_; };
Define the iterator for the DLL:
class DLLIterator { public: // takes in a node to mark the start of the iteration DLLIterator(Node *head) : curr_(head) {} // prefix increment operator (++iter) DLLIterator &operator++() { // must use -> to access the member of a pointer! // use . if accessing the object itself curr_ = curr_->next_; return *this; } // postfix increment operator (iter++) // the (int) is a dummy parameter to differentiate // the prefix and postfix increment DLLIterator operator++(int) { DLLIterator temp = *this; // this is a pointer to the current object // *this returns the iterator object // ++*this calls the prefix increment operator, // which equals to `this->operator++()` ++*this; return temp; } // implement the equality operator // an lvalue reference argument avoids the copy // the const in the parameter means this function // cannot modify the argument // the `const` outside the parameter list means // the function cannot modify `this` bool operator==(const DLLIterator &str) const { return itr.curr_ == this->curr_; } // implement the dereference operator to return the value // at the current iterator position int operator*() { return curr_->value_; } private: Node *curr_; };
Define DLL:
class DLL { public: DLL() : head_(nullptr), size_(0) {} // the destructor deletes nodes one by one ~DLL() { Node *current = head_; while (current != nullptr) { Node *next = current->next_; delete current; current = next; } head_ = nullptr; } // after the insertion `new_node` becomes the new head // `head` is just a pointer to the node void InsertAtHead(int val) { Node *new_node = new Node(val); // new_node->next points to the object pointed by head_ new_node->next_ = head_; if (head_ != nullptr) { head_->prev_ = new_node; } head_ = new_node; size_ += 1; } DLLIterator Begin() { return DLLIterator(head_); } // returns the pointer pointing one after the last element // used in the loop to determine whether the iteration ends // e.g., `for (DLLIterator iter = dll.Begin(); iter != dll.End(); ++iter)` DLLIterator End() { return DLLIterator(nullptr); } Node *head_{nullptr}; // in-class initializers size_t size_; };
4. STL containers
- The C++ STL (standard library) is a generic collection of data structure and algorithm implementations, e.g., stacks, queues, hash tables.
- Each container has own header, e.g.,
std::vector
. - The
std::set
is implemented as a red-black tree.
4.1. Vector
#include <algorithm> // to use std::remove_if #include <iostream> // to use std::cout #include <vector> // A helper class used for vector class Point { public: // constructors Point() : x_(0), y_(0) {} Point(int x, int y) : x_(x), y_(y) {} // inline asks the compiler to substitute the function // directly at the calling location instead of performing // a normal function call, to improve performance for small functions inline int GetX() const { return x_; } inline void SetX(int x) { x_ = x; } void PrintPoint() const { std::cout << "Point: (" << x_ << ", " << "y_" << ")\n"; } private: int x_; int y_; }; int main() { std::vector<Point> point_vector; // approach 1 to append to a vector point_vector.push_back(Point(35, 36)); // approach 2, pass the argument to Point(x,y) constructor // slightly faster than push_back point_vector.emplace_back(37, 38); // iterate through index // size_t: unsigned integers specifially used in loop or counting for (size_t i = 0; i < point_vector.size(); ++i) { point_vector[i].PrintPoint(); } // iterate through mutable reference for (Point &item : point_vector) { item.SetX(10); } // iterate through immutable reference for (const Point &item : point_vector) { item.GetX(); } // initialize the vector with an initializer list std::vector<int> int_vector = {0, 1, 2, 3}; // erase element given its index // int_vector.begin() returns a std::vector<int>::iterator // pointing to the first elemnt in the vector // the vector iterator has a plus iterator int_vector.erase(int_vector.begin() + 2); // {0, 1, 3} // erase a range of elements // int_vector.end() points to the end of a vector (not the last element) // and cannot be accessed. int_vector.erase(int_vector.begin() + 1, int_vector.end()); // {0} // erase elements via filtering // std::remove_if(range_begin, range_end, condition) returns an iterator // pointing to the first element to be erased // remove_if() also partitions point_vector so that unsatisfied elements are // moved before the point_vector.begin(), i.e., the vector is reordered point_vector.erase( std::remove_if(point_vector.begin(), point_vector.end(), [](const Point &point) { return point.GetX() == 10; }), point_vector.end()); return 0; }
4.2. Set
#include <iostream> #include <set> int main() { std::set<int> int_set; // can insert element with .insert() or .emplace() // .emplace() allows to construct the object in place for (int i = 1; i <= 5; ++i) { int_set.insert(i); } for (int i = 6; i <= 10; ++i) { int_set.emplace(i); } // iterate the set for (std::set<int>::iterator it = int_set.begin(); it != int_set.end(); ++it) { std::cout << *it << " "; } std::cout << "\n"; // .find(key) returns an iterator pointing to the key // if it is in the set, otherwise returns .end() std::set<int>::iterator search = int_set.find(2); if (search != int_set.end()) { std::cout << "2 is not found\n"; } // check whether the set contains a key with .count() // it returns either 0 or 1 as each key is unique if (int_set.count(11) == 0) { std::cout << "11 is not in the set.\n"; } // erase a key, returns the count of removed elements 0 or 1 int_set.erase(4); // erase a key given its position, returns the iterator to the next element int_set.erase(int_set.begin()); // erase a range of elements int_set.erase(int_set.find(9), int_set.end()); }
4.3. Unordered maps
#include <iostream> #include <string> #include <unordered_map> #include <utility> // to use std::make_pair int main() { std::unordered_map<std::string, int> map; // insert items map.insert({"foo", 2}); map.insert({{"bar", 1}, {"eggs", 2}}); // insert items via pairs map.insert(std::make_pair("hello", 10)); // insert items in an array-style map["world"] = 3; // update the value map["foo"] = 9; // .find() returns an iterator pointing to the item // or the end std::unordered_map<std::string, int>::iterator result = map.find("bar"); if (result != map.end()) { // one way to access the item // both '\n' and std::endl prints newliine, but std::endl // also flushes the output buffer, so use '\n' is better std::cout << "key: " << result->first << " value: " << result->second << std::endl; // another way is dereferencing std::pair<std::string, int> pair = *result; std::cout << "key: " << pair.first << " value: " << pair.second << std::endl; // check whether a key exists with .count() if (map.count("foo") == 0) { std::cout << "foo does not exist\n"; } // erase an item via a key map.erase("world"); // or via an iterator map.erase(map.find("bar")); // can iterate the map via iterator or via for-each for (std::unordered_map<std::string, int>::iterator it = map.begin(); it != map.end(); ++it) { std::cout << "(" << it->first << ", " << it->second << "), "; } std::cout << "\n"; } }
5. auto
auto
keyword tells the compiler to infer the type via its initialization expression.
#include <vector> #include<unordered_map> #include<string> #include<iostream> // create very long class and function template <typename T, typename U> class VeryLongTemplateClass { public: VeryLongTemplateClass(T instance1, U instance2) : instance1_(instance1), instance2_(instance2) {} private: T instance1_; U instance2_; }; template <typename T> VeryLongTemplateClass<T, T> construct_obj(T instance) { return VeryLongTemplateClass<T, T>(instance, instance); } int main() { auto a = 1; // a is int auto obj1 = construct_obj<int>(2); // can infer // auto defaults to copy objects rather than taking the reference std::vector<int> int_values = {1, 2, 3, 4}; // a deep-copy happens auto copy_int_values = int_values; // this creates a reference auto &ref_int_values = int_values; // use auto in the for loop is very common std::unordered_map<std::string, int> map; for (auto it = map.begin(); it != map.end(); ++it) { } // another exmaple std::vector<int> vec = {1, 2, 3, 4}; for (const auto &elem : vec) { std::cout << elem << " "; } std::cout << std::endl; }
6. Smart pointers
- A smart pointer is a data structure used in languages that do not have built-in memory management (e.g., with garbage collection, e.g., Python, Java) to handle memory allocation and deallocation.
std::unique_ptr
andstd::shared_ptr
are two C++ standard library smart pointers, they are wrapper classes over raw pointers.std::unique_ptr
retains sole ownership of an object, i.e., no two instances ofunique_ptr
can manage the same object, aunique_ptr
cannot be copied.std::shared_ptr
retains shared ownership of an object, i.e., multiple shared pointers can own the same object and can be copied.
6.1. std::unique_ptr
#include <iostream> #include <memory> // to use std::unique_ptr #include <utility> // to use std::move // class Point is the same as before // takes in a unique point reference and changes its x value void SetXTo10(std::unique_ptr<Point> &ptr) { ptr->SetX(10); } int main() { // initialize an empty unique pointer std::unique_ptr<Point> u1; // initialize a unique pointer with constructors std::unique_ptr<Point> u2 = std::make_unique<Point>(); std::unique_ptr<Point> u3 = std::make_unique<Point>(2, 3); // can treat a unique pointer as a boolean to determine whether // the pointer contains data std::cout << "Pointer u1 is " << (u1 ? "not empty" : "empty") << std::endl; if (u2) { std::cout << u2->GetX() << std::endl; } // unique_ptr has no copy constructor! std::unique_ptr<Point> u4 = u3; // won't compile! // can transfer ownership with std::move std::unique_ptr<Point> u5 = std::move(u3); // then u3 becomes empty! // pass the pointer as a reference so the ownership does not change SetXTo10(u5); // can still access u5 afterwards return 0; }
- Note that the compiler does not prevent 2 unique pointers from pointing to the same object.
// the following code can compile // but it causes a double deletion! MyClass *obj = new MyClass(); std::unique_ptr<MyClass> ptr1(obj); std::unique_ptr<MyClass> ptr2(obj);
6.2. std::shared_ptr
#include <iostream> #include <memory> #include <utility> // class Point is the same as before // modify a Point inside a shared_ptr by passing the reference void modify_ptr_via_ref(std::shared_ptr<Point> &point) { point->SetX(10); } // modify a Point inside a shared_ptr by passing the rvalue reference // If an object is passed by rvalue reference, one should assume the ownership // is moved after the function call, so the original object cannot be used void modify_ptr_via_rvalue_ref(std::shared_ptr<Point> &&point) { point->SetY(11); } // copy a shared_pointer with the default constructor // so one should define own copy constructor and assignment when an object // contains pointers void copy_shard_ptr_in_function(std::shared_ptr<Point> point) { std::cout << "Use count of the shared pointer: " << point.use_count() << std::endl; // add by 1 // the copy is destroyed at the end, so the count is decremented } int main() { // the pointer constructors are the same as unique_ptr std::shared_ptr<Point> s1; std::shared_ptr<Point> s2 = std::make_shared<Point>(); std::shared_ptr<Point> s3 = std::make_shared<Point>(2, 3); // copy a shared pointer via copy assignment or copy constructor // increment the reference count, which can be tracked by .use_count std::cout << "Use count of s3" << s3.use_count() << std::endl; // 1 std::shared_ptr<Point> s4 = s3; // copy assignment std::cout << "Use count of s3" << s3.use_count() << std::endl; // 2 std::shared_ptr<Point> s5(s4); std::cout << "Use count of s3" << s3.use_count() << std::endl; // 3 s3->SetX(100); // also changes the data in s4 and s5 std::shared_ptr<Point> s6 = std::move(s5); // s5 transfer the ownership to s6 std::cout << "Use count of s3" << s3.use_count() << std::endl; // still 3: s3, s4, s6 // as unique_ptr, shared_ptr can be passed by references and rvalue references modify_ptr_via_ref(s2); // setX(11) modify_ptr_via_rvalue_ref(std::move(s2)); // setY(12) std::cout << "s2.x = " << s2->GetX() << " , s2.y = " << s2->GetY(); // (11, 12) // shared_ptr can also be passed by value/copy std::cout << "Use count of s2" << s2.use_count() << std::endl; // 1 copy_shared_ptr_in_function(s2); // inside the function, the use count is 2 std::cout << "Use count of s2" << s2.use_count() << std::endl; // 1 again as the copy is detroyed return 0; }
7. Synchronization
7.1. std::mutex
#include <iostream> #include <mutex> #include <thread> // define a global variable to be modified // by multiple threads int count = 0; // declare a mutex std::mutex m; void add_count() { m.lock(); // acquire the lock count += 1; m.unlock(); // release the lock } int main() { std::thread t1(add_count); std::thread t2(add_count); t1.join(); t2.join(); std::cout << count << std::endl; // always 2 return 0; }
7.2. std::scoped_lock
- A mutex wrapper that provides a RAII-style of obtaining and releasing the lock.
- When the object is constructed, the locks are acquired; when the object is destructured, the locks are released.
- Better than
std::mutex
since it provides exception safety.
#include <iostream> #include <mutex> int count = 0; std::mutex m; void add_count() { // the scoped_lock constructor allows for the thread // to acquire the mutex m std::scoped_lock slk(m); count += 1; // once the function finishes, slk is destructurd and // m is released }
7.3. std::condition_variable
- Allow threads to wait until a particular condition before acquiring a mutex.
- Allow other threads to signal waiting threads to alert them that the condition may be true.
notify_one
#include <condition_variable> #include <iostream> #include <mutex> #include <thread> int count = 0; std::mutex m; // declare a condition variable std::condition_variable cv; void add_count_and_notify() { std::scoped_lock slk(m); count += 1; if (count == 2) { // notidy one waiting thread // otherwise the waiter thread hangs forever cv.notify_one(); } } void waiter_thread() { // a unique_lock is movable but not copiable // more flexible than scoped_lock as it can unlock // and relock manually, while scoped_lock can only be // unlocked automatically. // scoped_lock cannot be used with condition variables std::unique_lock lk(m); cv.wait(lk, [] { return count == 2; }); std::cout << count << std::endl; } int main() { std::thread t1(waiter_thread); // make t1 really waits std::this_thread::sleep_for(std::chrono::milliseconds(100)); std::thread t2(add_count_and_notify); std::thread t3(add_count_and_notify); t1.join(); t2.join(); t3.join(); return 0; }
7.4. Reader-writer lock
- A reader-writer lock allows multiple threads to have shared read access (no writers are allowed during read operations) and exclusive write access, i.e., no other readers or writers are allowed during the write.
- C++ does not have a specific reader-writer’s lock library, but can be emulated with
std::shared_mutex
,std::shared_lock
andstd::unique_lock
. std::shared_mutex
is a mutex that allows for both shared read-only locking and exclusive write-only locking.std::shared_lock
can be used as an RAII-style read lock.std::unique_lock
can be used a RAII-style exclusive write lock.#include <iostream> #include <mutex> #include <shared_mutex> #include <thread> int count = 0; // resource std::shared_mutex m; void read_value() { // use shared_lock to gain read-only, shared access std::shared_lock lk(m); std::cout << "Reading " + std::to_string(count) << std::endl; } void write_value() { // use unique_lock to gain exclusive access std::unique_lock lk(m); count += 3; } int main() { // since all 3 threads run in parallel, // the output is not deterministic std::thread t1(read_value); std::thread t2(write_value); std::thread t3(read_value); t1.join(); t2.join(); t3.join(); return 0; }
Build a free Telegram sticker tag bot
1. What happened
When I started to use Telegram, I really missed being able to search stickers by text, like I can in WeChat. Sadly, Telegram only lets you search emojis by text, or find stickers using emojis.
After digging around, I discovered the easiest way to add this cool feature was through Telegram bots. The idea? Store a tag-to-sticker map in the bot, then fetch all matching stickers when given a tag in the bot’s inline mode. There are a few sticker tag bots on Telegram already, but they’re either dead or can’t handle Unicode input like Chinese characters. Plus, I’m not super keen on trusting my data to someone else’s database. Moreover, I might want to use a bot for other personal stuff later.
So, I decided to build my own Telegram bot.
2. What do I need
My goal was to create a bot using only free services, including cloud storage for key-value pairs and a hosting platform to keep the bot running.
I stumbled upon Render from an X recommendation which offers 750 hours per month for free deployment (which equals 31 days), so I deployed my bot there once I got the bot running locally. But then I found out Render’s free tier doesn’t offer permanent storage and shuts down services after 15 minutes of inactivity.
A sticker tag bot without memory isn’t much use to anyone, so I went hunting for free cloud storage. With some help from Claude and Perplexity, I discovered Firebase Realtime database, which offers 1GB storage and 10GB throughput per month on its free tier.
Even with cloud storage, a bot that konks out every 15 minutes just won’t cut it - I need my stickers now! So my next quest was finding a way to keep the bot awake, which led me to UptimeRobot. It’s actually a web monitoring tool, but I can use it to ping the bot regularly, tricking it into staying “active” instead of dozing off.
So, to sum it up, building this sticker tag bot required:
- a Telegram bot from BotFather,
- a free Render deployment,
- a Firebase’s free key-value storage, and
- an UptimeRobot’s free monitoring service.
However, these services do not work together automatically. Gluing them together required additional effort.
3. How to build a bot
The first step in building any bot is asking BotFather for a new bot and keeping the bot token secure. Telegram offers a helpful tutorial that explains the process using Java. Examples in other languages can be found in their Gitlab repo. In my opinion, the most challenging part here is creating a unique bot username that is still available.
The next step involves working with the Telegram bot API in the chosen programming language. This includes learning how to handle messages effectively. For example, I used tgbotapi(Golang).
3.1. How to build a sticker tag bot
A sticker tag bot needs two main functionalities:
- Handle direct messages to add new sticker tags.
- Handle inline queries to search for stickers using a given tag.
To implement the first functionality, I created a simple state machine with two states:
- The initial state waits for a sticker input and then moves to the tag state.
- The tag state waits for any text input to use as the tag for the previous sticker.
To implement the second functionality, one needs to use the InlineQueryResultCachedSticker method.
For local testing, one can use a lightweight local key-value storage to store and search sticker tags. I used BadgerDB(Golang) for example.
I noticed that the generated file ID for the same sticker is different each time, making it hard to check for duplicates when adding new tags.
To address this, I added a /delete
method to remove tags when needed.
3.2. How to make the bot private
I couldn’t find an official way to make a bot visible only to me. Suggested by Claude, I predefined a list of authorized users. Then I performed a sanity check before handling any messages.
4. How to deploy the bot on Render
Deploying a service on Render with a free account is challenging due to the lack of shell access, disk access, and non-so-live logs. The process of making everything work at this stage was time-consuming and I even contacted Render’s technical support although they only responded “Ok I will close the ticket” after the issue was self-resolved.
Three main steps are required here:
- start an HTTP server with several endpoints at the bot, and
- configure the web service and environment variables on Render’s dashboard,
- configure the Telegram webhook at the bot.
In step 1, starting an HTTP server at 0.0.0.0:<some-port>
is necessary.
One should also enable GET methods for the root and a health check endpoint to allow Render to scan the service regularly.
In step 2, one needs to fill in the service configuration and environment variables in different boxes.
This includes settings such as port, build command, and health check endpoint.
The issue I encountered was Render could not scan any port even if I have triple-checked that everything worked fine locally.
In the end, I solved this issue by adding the Golang build tag -tags netgo
in the build command.
Actually this flag was configured by default, but I initially replaced it with a simpler build command.
In step 3, one needs to configure the webhook with the Bot API and to enable the POST method for the webhook at the HTTP server (this can also be handled by the Bot API).
The webhook can be https: //<your project>.onrender.com/<your-token>
(or another unique URL).
This URL informs Telegram where to send and receive all messages for the bot.
5. How to connect to the Firebase
The Firebase Realtime database stores key-value pairs in the JSON format. Connecting the bot with the database requires the following steps:
- Create the app and the database on Firebase’s dashboard.
Specifically, one needs to store the following 3 values for interaction:
- The database URL, which looks like
https: //<your-app>-default-rtdb.*.firebasedatabase.app
. - The credential file, which can be downloaded at
Project settings->Service accounts->Firebase Admin SDK
(and should also be added to Render).
- The database URL, which looks like
- Import the language-specific Firebase API to configure the database in the bot. For example, I use firebase(Golang).
- Update the database rules in Firebase dashboard to only allow authorized writes for specific tags, e.g., one name/path to refer to those key-value pairs.
It’s worth noting that connecting to the database on Render may take some time after a fresh start.
During this initialization period, the log may display a 502 Bad Gateway
error to the database.
6. How to configure UptimeRobot
Before configuring UptimeRobot, an attempt was made to ping the bot from within itself, but this approach did not function for Render.
Using UptimeRobot to maintain the bot’s active status involves two primary steps:
- Enable the HEAD method (the sole method available for a free account) for any endpoint on the HTTP server.
- Configure an HTTP(S) monitor for that endpoint, which appears as
<you-project>.onrender.com/<endpoint>
, and establish the monitoring interval to less than 15 minutes.
7. Conclusion
This post isn’t meant to be a step-by-step guide for building a Telegram bot. It skips some steps and doesn’t include screenshots. But don’t worry, most of the missing bits can be figured out using AI language models these days. The rest really depends on each specific situation. The main point here is to show how to set up a free small web service, even when there’s no single platform that does it all.
When I first wrote this, my bot had been up and running for 10 days. It only had 30 minutes of downtime, which I think happened because UptimeRobot couldn’t reach Render’s IP address during that time.
Right now, the repository is private since I plan to add a second functionality to the bot soon.