Notes on technology, startups, and life.https://frankk.site/en© 2026 Frankhttps://frankk.site/en/blog/business-taste/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/business-taste/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enPeter Thiel put in a lifeline when SpaceX was nearly out of money; by the day it went public, the stake was worth more than 50 billion dollars. From the monopoly philosophy of Zero to One to Girard's mimetic desire, Thiel's calls as an investor and a founder are really one and the same way of thinking about how to escape competition.Sat, 13 Jun 2026 00:00:00 GMTBusinessStartupshttps://frankk.site/en/blog/how-to-persuade/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/how-to-persuade/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThe best theory of persuasion comes not from a modern business school but from Aristotle, twenty-three hundred years ago: Ethos (credibility), Pathos (emotion), Logos (logic). Most popular persuasion frameworks today can be seen as localized versions of these three elements for different audiences. This piece gives a template you can apply to each of ten scenarios—Claim → Reason → Evidence for a raise, STAR for interviews, the elevator pitch, BLUF for email, Problem → Solution → Traction for the pitch, Pain → Gain → Proof for sales, the Golden Circle for the launch event, BATNA for negotiation, AIDA for copywriting, Nonviolent Communication for intimate relationships—walking through them one scenario, one template at a time, and landing on a single line: the framework is the map, sincerity is the compass.Fri, 12 Jun 2026 00:00:00 GMTEssayshttps://frankk.site/en/blog/us-sanctions-impact/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/us-sanctions-impact/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThree parts: the nine-tier intensity ladder of U.S. sanctions — what each tier does to you, with cases; three archetypal sanctioned countries (Iran, North Korea, Russia) and how ordinary people, officials and merchants fare in each; and 11 China-related cases — ZTE, Huawei, Jinhua, Bank of Kunlun, Carrie Lam, the three telecoms, Shandong teapot refineries, Tencent/CATL, and Shanghai Heiying's Zhou Shuai.Wed, 10 Jun 2026 00:00:00 GMTUS SanctionsOFACSDN ListEntity ListExport ControlsUSD ClearingUS-Chinahttps://frankk.site/en/blog/overseas-product-development/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/overseas-product-development/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enClaude Code has gotten really good lately, and a lot of people now have a shot at building their own products. This post shares the tech stack and the various SaaS platforms I commonly use when building products for overseas markets: static sites, full front-end/back-end apps, mobile, and desktop—covering framework choices, deployment, databases, auth, payments, AI services, and image hosting, with free-tier limits and starting costs wherever I can.Tue, 09 Jun 2026 00:00:00 GMTTechnologyBusinesshttps://frankk.site/en/blog/systematic-self-iteration/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/systematic-self-iteration/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enKarpathy on how to become an expert and Feynman on how to test real understanding are, at bottom, the same thing: use output to check input. This piece maps the method—do a project first, fill in theory later, only compare yourself to your past self—onto the training loop of a neural network. Forward is acting first, Loss is the gap from the ideal result, and Backward plus gradient descent is the review-and-improve step. The biggest difference between people and networks is that a large Loss hurts us, so the key is to separate criticizing behavior from negating worth: gradient descent only updates the parameters (your behavior, your skills), so don't blow up the architecture (who you are). Finally, set it against Ray Dalio's Principles: the very same iterative logic of Pain + Reflection = Progress, independently distilled into a system by a man who runs a hedge fund.Fri, 05 Jun 2026 00:00:00 GMTEssayhttps://frankk.site/en/blog/apple-ai-inference/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/apple-ai-inference/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enWith WWDC 2026 approaching, this note maps Apple's two local AI inference paths (Neural Engine vs. GPU + Neural Accelerator), traces compute and bandwidth evolution across recent silicon generations, and benchmarks them against Nvidia across four dimensions — compute, memory capacity, bandwidth, and precision. The verdict: Apple's edge is "local, power-efficient, and capable of running large models (especially MoEs)" — not peak throughput on dense large models.Wed, 03 Jun 2026 00:00:00 GMTAppleApple SiliconLocal InferenceNeural EngineMLXNvidiaMoEhttps://frankk.site/en/blog/go-concurrency-patterns/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/go-concurrency-patterns/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA ready-to-use Go concurrency reference. From "locking shared variables, Once, channels, Pub/Sub" to "select/Actor, pipeline/fan-out/worker pool, context, errgroup," each pattern gives when to use it, template code, and key pitfalls, closing with a cheat sheet: if you can avoid sharing, communicate instead; if you must share, wrap it with the simplest synchronization possible; and every goroutine needs a clear exit path.Wed, 03 Jun 2026 00:00:00 GMTGoConcurrencychannelgoroutinehttps://frankk.site/en/blog/data-center-power/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/data-center-power/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enAI has overnight made electricity the most expensive bottleneck in data centers. On one side is "where the power comes from" — from utility grids to behind-the-meter fuel cells, gas turbines, and nuclear; on the other is "where the power goes" — the breakdown across GPUs, other server components, cooling, and networking. This piece walks through the power system of this space from both the supply and demand sides.Mon, 01 Jun 2026 00:00:00 GMTdata centerpowerfuel cellgas turbinenuclearGPUcoolinghttps://frankk.site/en/blog/claude-code-practices/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/claude-code-practices/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enI've been using Claude Code for almost a year now. As one of its earliest users, I've watched both Claude Code and the models themselves improve enormously over the past year. This post shares some ways I use it beyond writing code, along with what I've learned: a workflow built on an always-on home machine + Tailscale + tmux; managing frontend and backend with a monorepo + submodules + OpenAPI; doing Deep Research with tree-structured prompts; cleaning up long conversations and messy desktop folders; and using it to draw SVGs, process images, make videos with Remotion, and more.Wed, 27 May 2026 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/storage-types-cpu-gpu/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/storage-types-cpu-gpu/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enWhy a large model generating text token by token is bottlenecked on memory bandwidth — starting from why the decoding phase is memory-bound, unpacking the GPU memory hierarchy from Pascal to Blackwell, and landing on the fabrication, cost, and supply chain of SRAM / GDDR / HBM, tracing one causal chain: bandwidth sets how fast the model can speak.Thu, 21 May 2026 00:00:00 GMTLLM推理GPU显存带宽HBMSRAMGDDR存储金字塔https://frankk.site/en/blog/hume-passion-as-hot-path/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/hume-passion-as-hot-path/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enIf you want to do something for the long haul, you have to love it. Through Hume's "reason is the slave of the passions" and a programmer's lens: emotion is the instruction set the CPU runs directly; reason is a high-level language that only works after layers of compilation. Doing something once, reason's long chain is fine; but anything you do long-term and at high frequency is a "hot loop," and only the lowest-cost driver—emotion, i.e. love—can keep it running, just as a hot loop can't be written in Python.Tue, 19 May 2026 00:00:00 GMTEssayshttps://frankk.site/en/blog/llm-low-precision-formats/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/llm-low-precision-formats/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA bit-level tour of the low-precision formats LLMs run on — first the representation basics for floating-point and integers, then FP32 / TF32 / FP16 / BF16 / FP8 / FP6 / FP4 / INT8 / INT4 one by one, then Qwen3.5’s release matrix as a real-world case of how models pick precision, and finally hardware support from Hopper to Blackwell.Mon, 18 May 2026 00:00:00 GMTLLMdata formatsquantizationFP8FP4INT4microscalingTensor CoreBlackwellQwen3https://frankk.site/en/blog/barbell-strategy/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/barbell-strategy/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enTaleb's barbell strategy is really two rules: push your resources toward the two ends and deliberately avoid the middle; and how you split between the ends depends on your situation—but never pile everything on one end. Its core was never about how much to put on each end, but about accepting that the world is fundamentally uncertain: first make sure you survive, then go after the opportunities that might let you leap.Sun, 17 May 2026 00:00:00 GMTEssayshttps://frankk.site/en/blog/hk-hiking-trails/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/hk-hiking-trails/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThe 100 km MacLehose Trail, Lion Rock sung about for fifty years, the pavilion at Pat Sin Leng that fire kept failing to consume—70% of Hong Kong is country park, and every trail threads through a piece of institutional history, an old song, or a war. 10 routes × history × culture.Sat, 09 May 2026 00:00:00 GMTHong KongHikingMacLehose TrailLion RockDragon's BackColonial HistoryWar Relicshttps://frankk.site/en/blog/consumer-hardware-research/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/consumer-hardware-research/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enHardcore gear that once belonged to defense, laboratories, and factories is being compressed by 2-3 orders of magnitude into out-of-the-box consumer products and moved into ordinary studies and living rooms — from DJI to Bambu to Hypershell, a systematic Shenzhen and Yangtze-Delta playbook.Fri, 27 Feb 2026 00:00:00 GMTDJIConsumer ElectronicsChinaRoboticsSupply Chainhttps://frankk.site/en/blog/social-media-business-model/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/social-media-business-model/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enContent creation may be the biggest lever available to ordinary people today—low barrier to entry, high ceiling, zero marginal cost of distribution. Looked at through the lens of business models, there are really only three ways to make a living from it: sell content directly, help others sell goods, or help yourself sell goodsSun, 15 Feb 2026 00:00:00 GMTBusinesshttps://frankk.site/en/blog/llm-inference-walkthrough/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/llm-inference-walkthrough/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA module-by-module breakdown of LLM inference: attention (MHA/MQA/GQA/MLA, linear attention, compression and sparsity), normalization, RoPE, gated FFN, and the core formulas of the prefill/decode phases — ending with the latest Qwen3.5 and DeepSeek-V4 as concrete architectural case studies.Thu, 12 Feb 2026 00:00:00 GMTLLMTransformerInferenceKV CacheGPUhttps://frankk.site/en/blog/tech-republic-book-review/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/tech-republic-book-review/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enPalantir CEO Alex Karp released a new book, The Technological Republic, in 2025, and it was published in mainland China at the end of the year. I bought it and read it right away. The views in the book represent the right-wing current of thought in Silicon Valley, and its shadow can be seen everywhere in actual American politics.Fri, 23 Jan 2026 00:00:00 GMTEssaysBusinesshttps://frankk.site/en/blog/surge-ai-data-supplier/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/surge-ai-data-supplier/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enI first heard of Surge AI from a podcast interview with Edwin Chen, right as they were raising their first round of funding. Edwin came across as extremely pragmatic and efficient, and his views left a strong impression.Sun, 21 Dec 2025 00:00:00 GMTBusinesshttps://frankk.site/en/blog/ui-tars-vision-language-model/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/ui-tars-vision-language-model/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enAccording to public reporting, the model used in the Doubao AI phone is a closed-source version of UI-TARS optimized for phones. UI-TARS itself is the result of SFT on top of Alibaba's Qwen2 VL, with the 7B version currently open-sourced (Qwen2 VL has open-sourced models from 3B to 72B). Rather than dwelling on Qwen here (Qwen2 VL already has UI Operation capabilities), this post focuses on how UI-TARS improves further on top of Qwen2 VL, split into data and training.Fri, 19 Dec 2025 00:00:00 GMTTechnologyBusinesshttps://frankk.site/en/blog/utm-tags-traffic-analysis/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/utm-tags-traffic-analysis/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enPromotion usually runs several channels at once: cold email, Google Ads, Twitter, SEO, communities. Without tagging your links, GA can't tell them apart. This post lays out the five UTM parameters clearly and gives a dozen real-world naming examples.Thu, 04 Apr 2024 00:00:00 GMTBusinesshttps://frankk.site/en/blog/jenni-ai-growth-strategy/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/jenni-ai-growth-strategy/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enJenni.ai is an AI tool for assisting with paper writing and reading. It has already reached \$5M ARR with 2.5M users and is still growing fast. In its early days it actually did SEO writing and could barely survive; later it narrowed down to academic writing and came back from the dead. CEO David Park has candidly shared their growth path from zero to several million dollars—from cold calls, to lurking in groups, to viral growth on TikTok—and much of it is worth learning from.Thu, 21 Mar 2024 00:00:00 GMTBusinesshttps://frankk.site/en/blog/startup-perks/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/startup-perks/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA systematic rundown of the perks the big international vendors offer startups, covering cloud, AI tokens, banking, and SaaS tools. With special attention to which ones you can apply for directly without a VC referral.Sun, 03 Mar 2024 00:00:00 GMTBusinesshttps://frankk.site/en/blog/shuangyue-bay/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/shuangyue-bay/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enJust two hours from Shenzhen, Shuangyue Bay in Huizhou is a place made for doing nothing: watch the sunrise over the water from your guesthouse balcony, set off fireworks and grill skewers on the beach, eat seafood far cheaper than in Shenzhen, then climb the lookout to see how two curving coastlines come together to form a "double moon." Includes a concrete two-day, one-night itinerary.Sat, 15 Apr 2023 00:00:00 GMTTravelhttps://frankk.site/en/blog/hong-kong-country-trails/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/hong-kong-country-trails/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enHong Kong is more than Victoria Harbour and Temple Street. Cross the border from Shenzhen and in half a day you can step into its other side—weathered ridges, alpine meadows like Wugong Mountain, deserted beaches all to yourself, and a coast of ten thousand columns left behind by Jurassic volcanoes. This piece gathers four country hikes: Beiling Double Crossing, Kai Kung Leng, Sharp Peak, and Po Pin Chau.Sat, 11 Feb 2023 00:00:00 GMTTravelhttps://frankk.site/en/blog/zhongshan-jiangmen/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/zhongshan-jiangmen/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enStringing Zhongshan and Jiangmen into a single weekend route: Thirty-Three Market Street recreates the old factory district from "The Knockout," the Mei Family Mansion is the Goose Town of "Let the Bullets Fly," a single ticket to the Zhongshan Film Studio lets you shoot scenes from many countries, plus two easy hikes up Shitou Mountain and Yaji Mountain. A hands-on guide for movie fans and photo lovers.Sat, 10 Dec 2022 00:00:00 GMTTravelhttps://frankk.site/en/blog/dapeng-peninsula/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/dapeng-peninsula/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enRight next to a megacity like Shenzhen, there's this pristine landscape of mountains and sea. Dongxichong, Dayanding, Daluqiang, the Mermaid Cave at Luzui Villa, and the Shenzhen Observatory—five spots on the Dapeng Peninsula I keep coming back to, from hardcore hikes to gentle seaside strolls you can take your parents on.Sat, 12 Nov 2022 00:00:00 GMTTravelhttps://frankk.site/en/blog/shenzhen-city-weekends/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/shenzhen-city-weekends/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enNo need to travel far—right within the city, Shenzhen hides plenty of places worth a visit: chase the sunset up Tanglang Mountain after work, watch the sun sink below the horizon at the Mangrove Park, listen to live music at sea aboard the Greater Bay Area No. 1 cruise, pray at Hongfa Temple and stroll through Fairy Lake Botanical Garden, or hike Meishajian at night to overlook Yantian Port.Sat, 15 Oct 2022 00:00:00 GMTTravelhttps://frankk.site/en/blog/wugong-mountain/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/wugong-mountain/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enWugong Mountain is in Pingxiang, Jiangxi, and is best known for its rolling alpine meadows—green grasslands in summer, with a better chance of catching a sea of clouds in winter. This is a record of a three-day, two-night trip from Shenzhen: taking the cable car up to skip the thousand-meter climb, walking the cliffside boardwalk, watching the sunrise, spending a night on the summit, and happening to catch a bonfire music festival on the meadow.Sat, 13 Aug 2022 00:00:00 GMTTravelhttps://frankk.site/en/blog/ncnn-peak-memory-mobilenet/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/ncnn-peak-memory-mobilenet/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enTo understand how much peak memory a model actually consumes during inference, I ran a layer-by-layer benchmark of MobileNet with NCNN on x86, compared several networks side by side, and analyzed how light_mode, fp16, and int8 affect peak memory — along with the differences in how "peak memory" itself is defined.Sat, 12 Jun 2021 00:00:00 GMTModel CompressionInference EngineMobileNethttps://frankk.site/en/blog/ml-system-design-overview/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/ml-system-design-overview/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enCourse notes from Stanford's CS 329S, Machine Learning Systems Design: inference and learning paradigms, data storage and feature management, sampling and class imbalance, parallel training and system testing, experiment-management tooling, and model deployment — a full picture of taking ML systems into production.Sun, 18 Apr 2021 00:00:00 GMTMLSysSystem DesignDeep Learninghttps://frankk.site/en/blog/deep-learning-profiling/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/deep-learning-profiling/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enPerformance bottlenecks in training and inference often aren't obvious; you need profiling to analyze them quantitatively. Notes on the profiling methods I use for the training phase (breaking down the time spent on data loading, the forward pass, and the backward pass) and the inference phase.Mon, 12 Apr 2021 00:00:00 GMTProfilingPerformance OptimizationDeep Learninghttps://frankk.site/en/blog/model-pruning/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/model-pruning/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enPruning is subtraction applied to a trained network. It breaks down into three steps: first decide how much to prune in each layer (by intuition, by sensitivity analysis, or by searching with reinforcement learning as in AMC), then actually prune (unstructured weight sparsity and the "Lottery Ticket Hypothesis"; or structured pruning that removes whole filters using criteria like L1/L2 norm, geometric median FPGM, BN's γ, or feature-map rank HRank), and finally finetune to recover the accuracy. Or you can simply prune while training—Slimmable Networks and AutoSlim.Sun, 07 Feb 2021 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/training-tricks/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/training-tricks/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA few scattered but practical training tricks. Reparameterization (ACNet / RepVGG / RepMLP) adds extra branches at training time to help optimization, then fuses them back into the original structure at inference—essentially gaining a bit of accuracy for free without adding any inference cost. Plus weight EMA, Shake-Shake, label smoothing, and Dropout—common tactics for fighting overfitting and improving generalization.Mon, 01 Feb 2021 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/knowledge-distillation/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/knowledge-distillation/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enKnowledge distillation is a common way to boost a small network's accuracy and speed up convergence, and it's often used to fine-tune small models after compression or NAS. The core idea is straightforward: treat the outputs of a high-accuracy "teacher network" as soft labels for a "student network," letting the student mimic the teacher. This post walks through several representative methods—classic temperature-scaled distillation, FitNets' intermediate-layer hints, FSP matrices, the teacher-assistant network (TAKD), and DML, where two networks learn from each other.Sat, 30 Jan 2021 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/data-augmentation/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/data-augmentation/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enData augmentation is a standard trick for preventing overfitting in computer vision. This post first walks through the common image augmentations—random resized crop, cutout, random erase, mixup—then covers the automated AutoAugment and Fast AutoAugment (I used the latter to win the image track of the AutoDL competition back in the day), and finally adds a note on test-time augmentation (TTA).Sun, 24 Jan 2021 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/class-imbalance/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/class-imbalance/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enReal-world data is rarely as tidy as public datasets, and severely imbalanced positive and negative samples are the norm. There are three main approaches to class imbalance: resampling (undersampling the majority class, e.g. Tomek Links; oversampling the minority class, e.g. SMOTE), weighting different classes or easy/hard samples (Focal Loss), and ensembles (Bagging).Mon, 18 Jan 2021 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/hyperparameter-tuning/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/hyperparameter-tuning/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enTuning in deep learning is both a black art and a make-or-break step—so much so that training is jokingly called "alchemy" and engineers call themselves "tuning monkeys." This post first covers the essentials of manual tuning—how batch size and learning rate move together, the update rules for SGD / momentum / Adagrad / Adam, learning-rate warmup and decay, and a small weight-decay trick—then turns to automated hyperparameter optimization (HPO): grid/random search, the CMA-ES evolutionary algorithm, and the most sample-efficient Bayesian optimization (Gaussian process + acquisition function), plus off-the-shelf tools like NNI and RAY.Sat, 12 Dec 2020 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/compact-network-design/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/compact-network-design/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enTo run on mobile with low latency and a small memory footprint, a family of compact networks was designed. This piece walks through their design ideas: MobileNet splits the standard convolution into depthwise + 1×1 convolutions (cutting FLOPs by up to 9×); ShuffleNet's grouped 1×1 convolution + channel shuffle; MobileNetV2's inverted residual; ShuffleNetV2's four design guidelines built around memory access cost (MAC); and MobileNetV3's h-swish, EfficientNet's compound scaling, and GhostNet's cheap feature generation.Sun, 06 Dec 2020 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/cnn-architecture-classics/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/cnn-architecture-classics/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enFollowing the ILSVRC timeline, this walks through several milestones in convolutional neural network architecture: LeNet-5 from 1998, AlexNet that ignited deep learning in 2012, GoogLeNet with its multi-branch Inception modules, the deeper VGG, Inception with Batch Normalization, ResNet that trained hundreds of layers via residual connections, and SENet that introduced channel attention.Tue, 01 Dec 2020 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/measuring-dl-performance/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/measuring-dl-performance/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enOptimization presupposes measurement—if you can not measure it, you can not improve it. A neural network's performance actually splits into two lines: task-oriented accuracy (Top-1 for classification, mAP for detection, mIoU for segmentation, PSNR for restoration) and efficiency-oriented cost (latency, FLOPs, parameter count, peak memory). This piece lays out the definitions and computations of these metrics, and why FLOPs so often lie (DenseNet161 is more than twice as slow as VGG16), all in one go.Sun, 15 Nov 2020 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/pytorch-training-acceleration/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/pytorch-training-acceleration/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enStarting from a baseline, this article progressively optimizes training speed through a variety of software and hardware methods, ultimately cutting training time to one eighth.Tue, 03 Nov 2020 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/auto-data-augmentation-survey/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/auto-data-augmentation-survey/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enNotes on three automated data augmentation surveys: AutoAugment searches for augmentation policies with reinforcement learning, RandAugment drastically shrinks the search space, and Fast AutoAugment speeds up the search by merging policies.Sat, 29 Feb 2020 00:00:00 GMTPaper NotesData AugmentationAutoMLhttps://frankk.site/en/blog/autodl-image-competition-tuning/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/autodl-image-competition-tuning/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA running log of tuning experiments on the AutoDL image-classification track (Pedro dataset): ResNet18 / 34 backbones, CELU in place of ReLU, resetting BN and Group Norm, FP16 mixed precision, ReID / pedestrian-attribute pretraining, and a series of controlled experiments including online-data reshuffling.Tue, 31 Dec 2019 00:00:00 GMTHyperparameter TuningImage ClassificationExperiment Loghttps://frankk.site/en/blog/nas-milestones/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/nas-milestones/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enNeural Architecture Search (NAS) has been red-hot this year. This post is a quick rundown of the work I personally find most representative. Corrections and additions are welcome, hahaha.Sun, 01 Dec 2019 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/deep-learning-gpu-feeding/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/deep-learning-gpu-feeding/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enI trained a lot of models recently and found that brute force is not always magic—more GPUs is not always better. Sometimes one V100 and two V100s make almost no difference, because the bottleneck is elsewhere. Here is a write-up of a few small tricks I have picked up.Mon, 12 Aug 2019 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/cvpr2019-pruning-notes/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/cvpr2019-pruning-notes/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThe core ideas of three CVPR 2019 pruning papers—Variational Pruning estimates channel saliency with a probability distribution, Importance Estimation, and Cascaded Projection for end-to-end compression and acceleration—plus the day's experimental progress.Wed, 10 Jul 2019 00:00:00 GMTPaper NotesPruningCVPRhttps://frankk.site/en/blog/progressive-pruning/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/progressive-pruning/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enMainstream pruning is a three-stage train-prune-finetune pipeline. These notes work through the idea of progressive pruning: using relative cross-layer statistics to guide non-uniform pruning, and treating the number of channels pruned each round as an analog of the learning rate, converging gradually rather than deciding all at once.Tue, 09 Jul 2019 00:00:00 GMTPruningModel CompressionIdeashttps://frankk.site/en/blog/dynamic-network-inference/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/dynamic-network-inference/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enStarting from a sensitivity-based pruning experiment, some thoughts on designing dynamic inference networks: why FLOPS is unreliable, abandoning multi-network dynamics in favor of single-network dynamics, attaching a classifier after each block for early exit, and how dynamic channel selection can be combined with pruning.Mon, 01 Jul 2019 00:00:00 GMTPruningDynamic InferenceModel Compressionhttps://frankk.site/en/blog/model-pruning-survey/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/model-pruning-survey/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enOrganizing a batch of model-pruning papers into three categories—predefined structured (L1, ThiNet, LASSO), automatic structured (Network Slimming, SSS), and others (AOFP, NISP, SNIP, Autopruner)—and discussing the thought-provoking conclusion that "for a predefined structure, training from scratch is enough."Thu, 23 May 2019 00:00:00 GMTPaper NotesPruningModel Compressionhttps://frankk.site/en/blog/mobilenetv3-lottery-ticket/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/mobilenetv3-lottery-ticket/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enTwo notes: MobileNetV3 applies NetAdapt fine-tuning on top of the MnasNet seed architecture, plus head/tail and activation-function tweaks — engineering refinements; the Lottery Ticket Hypothesis argues that large networks hide trainable sparse sub-networks, contrasted with "Rethinking the Value of Network Pruning."Thu, 09 May 2019 00:00:00 GMTPaper NotesMobileNetPruninghttps://frankk.site/en/blog/dorefa-net-low-bit-quantization/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/dorefa-net-low-bit-quantization/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enDoReFa-Net quantizes weights, activations, and gradients separately to low bit-width, pushing most of the computation in both training and inference down to the bit-operation level. From the relationship between bit operations and dot products, to the STE (Straight-Through Estimator), to how each of the three is quantized, the overall algorithm flow, and the fused inference optimization.Mon, 06 May 2019 00:00:00 GMTPaper NotesQuantizationModel Compressionhttps://frankk.site/en/blog/pix2pix-image-translation/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/pix2pix-image-translation/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enpix2pix uses a conditional GAN for general-purpose image translation—the same architecture transfers across tasks just by swapping datasets. Why not use L1 / L2 directly (they only capture low frequencies and produce blurry results), how PatchGAN models high frequencies, noise inputs and stochasticity, and the role of BN.Wed, 24 Apr 2019 00:00:00 GMTPapersGANImage Translationhttps://frankk.site/en/blog/gan-training-techniques/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/gan-training-techniques/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enNotes on "Improved Techniques for Training GANs": GAN training is essentially the search for a Nash equilibrium in a high-dimensional non-convex game, which makes convergence hard. A walkthrough of feature matching, minibatch discrimination, historical averaging, one-sided label smoothing, virtual BN, plus the Inception Score and the semi-supervised setup.Tue, 23 Apr 2019 00:00:00 GMTPaper NotesGANTraining Techniqueshttps://frankk.site/en/blog/nas-pruning-ideas/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/nas-pruning-ideas/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA batch of scattered ideas accumulated in NAS and model compression, spanning search strategies (MCMC, greedy linear structures), pruning, dynamic inference, and adaptive inference, gathered into a pick-and-experiment checklist, with reading notes on the AutoML Survey.Thu, 18 Apr 2019 00:00:00 GMTNASModel CompressionIdeashttps://frankk.site/en/blog/octave-conv/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/octave-conv/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enOctaveConv's angle isn't parameter count but the redundancy in feature maps—decompose them into low- and high-frequency components, control the ratio with a hyperparameter α, and store and compute the low-frequency part at a lower resolution. The parameter count stays the same while the compute drops.Tue, 16 Apr 2019 00:00:00 GMTPapersModel CompressionConvolutionhttps://frankk.site/en/blog/cv-attention-mechanisms-comparison/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/cv-attention-mechanisms-comparison/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA side-by-side comparison of several attention mechanisms in CV: SENet weights channels, Non-local applies self-attention over spatial pixels, CBAM chains the two, DANet uses dual attention for segmentation, plus notes on how Single-Path NAS compresses multi-path search into a single path.Thu, 11 Apr 2019 00:00:00 GMTPapersAttentionNAShttps://frankk.site/en/blog/resource-constrained-nas-submodular/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/resource-constrained-nas-submodular/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enSearching for the highest-accuracy network under a resource budget is NP-hard. This paper casts NAS into the submodular optimization framework from combinatorial optimization, obtaining an approximation guarantee via a greedy algorithm and further cutting the compute cost—explaining, in theory, why greedy search is reasonable.Wed, 10 Apr 2019 00:00:00 GMTPaper NotesNASOptimization Algorithmshttps://frankk.site/en/blog/randomly-wired-neural-networks/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/randomly-wired-neural-networks/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enKaiming He et al. use random graph generators (ER / BA / WS) to produce a network's wiring directly, skipping selection altogether—under the same generator parameters, the performance variance across different random seeds is tiny. Notes on the three random-graph algorithms, the human priors retained in the overall network structure, and the ImageNet / COCO / robustness experiments.Sun, 07 Apr 2019 00:00:00 GMTPaper NotesNASNetwork Architecturehttps://frankk.site/en/blog/deep-learning-interpretability-survey/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/deep-learning-interpretability-survey/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA set of interpretability notes built around "Visual Interpretability for Deep Learning: a Survey," walking through visualization, representation diagnosis, decomposing networks into explanatory graphs/decision trees, learning interpretable representations directly, and evaluation metrics — with Interpretable CNN, Network Dissection, and a black-box explanation survey thrown in.Mon, 01 Apr 2019 00:00:00 GMTPaper NotesInterpretabilityCNNDeep Learninghttps://frankk.site/en/blog/generative-models-and-training-tricks/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/generative-models-and-training-tricks/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enTwo sets of notes combined: one on the basic landscape of generative models (PixelRNN / CNN, AE / VAE, GAN), and one a checklist of training tricks for image classification (training, data augmentation, validation). They sit together because the regularization and stability ideas in generative models share a lot with discriminative training.Sat, 30 Mar 2019 00:00:00 GMTGenerative ModelsGANTraining Trickshttps://frankk.site/en/blog/dataset-distillation/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/dataset-distillation/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enDataset distillation compresses an entire dataset into a few synthetic images per class plus a learning rate, reaching near full-training accuracy in just a few iterations. We work through the paper's five stages step by step: fixed / random initialization, linear-model analysis, multi-step gradients, and different initialization distributions.Thu, 28 Mar 2019 00:00:00 GMTPapersDataset DistillationModel Compressionhttps://frankk.site/en/blog/auto-deeplab-nas-search-space/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/auto-deeplab-nas-search-space/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enAuto-DeepLab moves NAS from classification to semantic segmentation, and beyond searching the internal structure of a cell, it also searches the macro network topology outside the cell. I use this paper to reflect on the essence of the NAS search space, and on one possible research direction.Thu, 21 Mar 2019 00:00:00 GMTPaper NotesNASSemantic Segmentationhttps://frankk.site/en/blog/tensorrt-inference-acceleration/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/tensorrt-inference-acceleration/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA quick rundown of what TensorRT is for and the key optimizations it uses to speed up inference: layer fusion, automatic kernel selection, low precision, and the rough flow of building an engine.Wed, 20 Mar 2019 00:00:00 GMTTensorRTInference AccelerationModel DeploymentDeep Learninghttps://frankk.site/en/blog/resnet-bag-of-tricks/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/resnet-bag-of-tricks/?utm_source=rss&utm_medium=rss&utm_campaign=rss_en"Bag of Tricks for Image Classification" stacks a pile of training tricks together, pushing ResNet-50's Top-1 on ImageNet from 75.3 to 79.29. A chapter-by-chapter record of the baseline pipeline, the hardware-oriented large batch / warm-up / FP16 tricks, plus cosine decay, label smoothing, knowledge distillation, mixup, and other refinements.Tue, 19 Mar 2019 00:00:00 GMTPaper NotesTraining TricksResNethttps://frankk.site/en/blog/mobilenetv2-notes/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/mobilenetv2-notes/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA close reading of MobileNetV2: why ReLU is dropped at the low-dimensional bottleneck, the inverted residual structure, choosing the expansion rate, the overall network and ReLU6 — all about balancing accuracy, FLOPs, and latency.Tue, 12 Mar 2019 00:00:00 GMTPaper NotesMobileNetNetwork Architecturehttps://frankk.site/en/blog/darts-proxylessnas-notes/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/darts-proxylessnas-notes/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enDARTS uses a softmax to relax the discrete choice of operations into a continuous mixture, then takes gradients with respect to the architecture parameters. To address its memory bottleneck, ProxylessNAS switches to binarized path sampling, searches directly on ImageNet, and models hardware latency as a differentiable objective. Reading the two together makes clear how differentiable NAS evolved from "saving search cost" to "saving memory and fitting the hardware."Sat, 23 Feb 2019 00:00:00 GMTPapersNASDeep Learninghttps://frankk.site/en/blog/paper-push-by-grasping/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/paper-push-by-grasping/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enEnd-to-end learning frameworks have become popular in robotic control: they take state/images as direct input and directly output predicted torques and action parameters. But they have been criticized for their heavy data requirements, sparking debate about their scalability—does the end-to-end approach require building a separate model for every task? Intuitively, sharing across tasks should help, since they all require some common understanding of the environment. This paper attempts the next step for data-driven end-to-end learning frameworks: moving from task-specific models to a joint model across multiple robotic tasks, with surprising results. Under the same amount of data, multi-task learning outperforms single-task learning. For the grasp task, for example, a model trained on 2.5k grasp samples plus 2.5k push samples outperforms a model trained on 5k grasp samples.Thu, 22 Nov 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/paper-dqn/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/paper-dqn/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThis paper by Volodymyr Mnih, published at NIPS 2013, is roughly the founding work of DQN; the other one is the Nature 2015 paper.Sat, 17 Nov 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/cityscapes-dataset/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/cityscapes-dataset/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enCityscapes is commonly used for semantic segmentation. Its data is divided into 8 categories in total, including one named "void", and each category contains multiple classes. Cityscapes has 30 classes in all, but once labeled there are 35 kinds of labels in total, which also include labels such as "unlabeled" that are not counted as classes.Fri, 02 Nov 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/paper-edanet/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/paper-edanet/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enEarlier networks for segmentation were either too slow or too inaccurate. This paper designs an EDANet module that combines asymmetric convolution, dilated convolution, and dense connectivity. It outperforms FCN across the board, and does so without a decoder structure, a context module, a post-processing scheme, or a pretrained model. Experiments are run on Cityscapes and CamVid. Tue, 30 Oct 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/paper-darts/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/paper-darts/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThis paper takes on architecture search by formulating the task in a differentiable form, instead of the traditional approach of using reinforcement learning over a discrete, non-differentiable space. The method is based on a continuous relaxation of the architecture representation, allowing efficient methods such as gradient descent to be used for architecture search. Subsequent experiments show that the algorithm performs well at discovering high-performance CNN architectures for image recognition and RNN architectures for language modeling, and is far faster than existing state-of-the-art non-differentiable architectures.Mon, 22 Oct 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/paper-hashing-tricks/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/paper-hashing-tricks/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enAs deep networks are increasingly deployed on mobile devices, a dilemma becomes ever more apparent: the trend in deep learning is to develop models that can absorb larger and larger datasets, yet mobile devices have limited storage and cannot hold overly large models. This paper proposes HashedNets, which reduce model size by exploiting the inherent redundancy inside neural networks. HashedNets use a low-cost hash function to randomly group connection weights into different hash buckets, and all connections that fall into the same bucket share a single parameter value. These parameters are tuned during standard backpropagation, and the hashing process introduces no extra memory overhead. Performance on a range of benchmark datasets shows that HashedNets can substantially reduce storage requirements while preserving generalization performance.Mon, 15 Oct 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/paper-shufflenet-v2/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/paper-shufflenet-v2/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enMany network designs today consider only indirect metrics of computational complexity (such as FLOPs), yet direct metrics (such as speed) are not determined by FLOPs alone—MAC (memory access cost) and platform characteristics also influence speed. This paper argues for measuring directly on a specific platform, which is far better than considering FLOPs alone. Based on a series of controlled experiments, it proposes several guidelines for efficient networks, and from those guidelines derives a new architecture, ShuffleNetV2. Comprehensive ablation experiments show the model achieves a state-of-the-art trade-off between performance and accuracy.Thu, 11 Oct 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/paper-shufflenet/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/paper-shufflenet/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThis paper introduces a highly efficient network, ShuffleNet, which centers on two operations—pointwise group convolution and channel shuffle—that drastically cut computation while maintaining accuracy. It outperforms prior networks on both ImageNet and COCO.Wed, 10 Oct 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/paper-mobilenets/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/paper-mobilenets/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enTargeting mobile and embedded vision applications, this paper proposes an efficient model called MobileNets, a lightweight neural network built on depthwise separable convolutions. The model uses two hyperparameters to trade off accuracy against latency, and extensive experiments balancing the two were conducted on ImageNet, demonstrating strong performance compared with other models. Experiments also showcase MobileNets' strengths across a wide range of applications, including object detection, fine-grained classification, face attributes, and large-scale geolocalization.Thu, 04 Oct 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/paper-inception-v4/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/paper-inception-v4/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enIn recent years, very deep convolutional neural networks have been the single biggest driver of improvements in image recognition performance. The Inception architecture achieves strong performance while keeping computational cost relatively low. Combining residual connections with conventional architectures produced the best results on the 2015 ILSVRC, comparable to Inception-V3. This paper considers combining Inception networks with residual connections; there is ample evidence that residual connections can greatly accelerate the training of Inception networks, and also evidence that a residual Inception slightly outperforms a non-residual Inception of almost the same computational cost. The paper also proposes several new Inception networks, both with and without residual connections, and these changes likewise markedly improve single-frame classification performance on the 2012 ILSVRC. Finally, it shows that scaling the activations appropriately can make the training of very wide residual Inception networks more stable.Fri, 28 Sep 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/matrix-vector-derivatives/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/matrix-vector-derivatives/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enMachine learning algorithms involve a great deal of matrix-related differentiation and derivatives. Here we introduce some common derivative formulas for matrices and vectors.Fri, 21 Sep 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/leetcode-stock-trading/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/leetcode-stock-trading/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThere is a class of dynamic-programming problems that give you a sequence of stock prices and ask for the maximum profit you can earn by buying and selling. These problems come in many variants — only one transaction allowed, unlimited transactions, an added transaction fee, and so on. In other words, the maximum profit is generally determined by the trading day and the maximum number of transactions allowed (where one transaction is a single buy paired with a single sell).Fri, 14 Sep 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/convex-sets/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/convex-sets/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enSimilar to solving the equality-constrained optimization problems discussed earlier, optimization problems with inequality constraints can likewise be solved using the method of Lagrange multipliersFri, 31 Aug 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/svm-derivation-part-3/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/svm-derivation-part-3/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThe previous posts covered the derivation of the hard-margin SVM. This post continues with the mathematical derivation of the soft-margin SVM, which allows some samples to be misclassified when the data is not linearly separable.Fri, 24 Aug 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/svm-derivation-part-2/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/svm-derivation-part-2/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enIn the previous post (Part 1) we discussed the derivation of the hard-margin SVM and its dual form, whose dual problem can be simplified into the following formSat, 18 Aug 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/svm-derivation-part-1/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/svm-derivation-part-1/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThe SVM is a classic method in machine learning. Beyond the hard-margin SVM, there are also variants such as the soft-margin SVM and the kernel trick. This article focuses on deriving the hard-margin SVM.Fri, 10 Aug 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/linear-equations-part-3/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/linear-equations-part-3/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThe pseudoinverse introduced here is the Moore-Penrose inverseThu, 26 Jul 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/linear-equations-part-2/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/linear-equations-part-2/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThe previous post covered one case of linear systems—where the number of unknowns is smaller than the number of equations—and introduced the least-squares method. This post covers the other case, where the number of equations is smaller than the number of unknowns. Here the system has infinitely many solutions, but the one closest to the origin—the solution with the smallest norm—is unique. This is the minimum-norm solution of a linear system that we introduce here.Sat, 21 Jul 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/leetcode-course-schedule/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/leetcode-course-schedule/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThis problem uses DFS and BFS to determine whether a given graph admits a topological ordering.Fri, 20 Jul 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/linear-equations-part-1/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/linear-equations-part-1/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThis article discusses solving one particular case of systems of linear equations, namely considering the systemFri, 20 Jul 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/ml-numerical-computation/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/ml-numerical-computation/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enMachine learning algorithms usually require a great deal of numerical computation—that is, solving for approximate values iteratively rather than obtaining analytical solutions. These algorithms typically involve optimization and solving systems of linear equations. Representing various floating-point numbers with a finite number of bits on a computer carries inherent error, so we need certain methods to guarantee the precision of our computations.Sat, 14 Jul 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/tensorflow-simple-neural-network/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/tensorflow-simple-neural-network/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enHere we use TensorFlow's Eager Execution to build the model, so that we no longer have to create a Graph and Session as before, which makes training a neural network more convenient and faster. Below we use the Iris dataset as an example to train a neural network, with the code taken from Google's tutorial.Fri, 06 Jul 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/deep-learning-on-geekcloud/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/deep-learning-on-geekcloud/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enI was recently working on an image-related deep learning assignment from my professor. After debugging the code, I found my computer didn't have enough memory (an 8GB laptop), and later I discovered a really handy deep learning cloud service platformFri, 29 Jun 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/kitti-lidar-camera-fusion/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/kitti-lidar-camera-fusion/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enKITTI offers many datasets; here we pick the raw_data (raw data) for fusion.Fri, 15 Jun 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/optimization-inequality-constraints/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/optimization-inequality-constraints/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enSimilar to the equality-constrained optimization problems discussed earlier, optimization problems with inequality constraints can also be solved using the method of Lagrange multipliers.Fri, 08 Jun 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/cpp-constructors/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/cpp-constructors/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enEvery class defines how its objects are initialized. A class controls the initialization of its objects through one or more special member functions called constructors. The job of a constructor is to initialize the data members of a class object, and a constructor runs whenever an object of the class is created.Sat, 02 Jun 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/cpp-associative-containers/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/cpp-associative-containers/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enAssociative containers support efficient lookup and access by key. The two primary associative containers are set and map. The elements of a map are key-value pairs, where the key acts as an index and the value represents the data associated with that index; the elements of a set contain only a key. A set supports efficient key lookup, and is presumably implemented with a hash table under the hood.Fri, 01 Jun 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/neural-network-backpropagation/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/neural-network-backpropagation/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enFor the training process of a neural network, the backpropagation algorithm lies at its coreFri, 01 Jun 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/cpp-sequential-containers/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/cpp-sequential-containers/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA container is a collection of objects of a specific type. Sequential containers give you control over the order in which elements are stored and accessed.Fri, 25 May 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/decision-tree-random-forest/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/decision-tree-random-forest/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA decision tree is a method for classification and regression; this article focuses mainly on decision trees used for classification. A decision tree has a tree-like structure, and in classification problems it represents the process of classifying data based on features. It can usually be regarded as a collection of if-then rules, or as a conditional probability distribution defined over the feature space and the class space. Its main advantages are good model interpretability and fast classification. During training, the decision tree model is built from the training data according to the principle of minimizing a loss function. During prediction, the decision tree is used to classify new data. Learning a decision tree usually involves three steps: feature selection, decision tree generation, and decision tree pruning. These decision-tree ideas come mainly from the ID3 algorithm proposed by Quinlan in 1986 and the C4.5 algorithm proposed in 1993, as well as the CART algorithm proposed by Breiman et al. in 1984.Thu, 24 May 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/cpp-io/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/cpp-io/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enC++ does not handle input and output directly; instead, it relies on a set of types defined in the standard library to deal with IO. These types support IO operations that read data from devices and write data to devices, where a device can be a file, a console window, and so on. Some types also allow in-memory IO, that is, reading data from a string, writing data to a string, and the like.Fri, 18 May 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/optimization-equality-constraints/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/optimization-equality-constraints/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enThis article discusses optimization problems of the following formFri, 18 May 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/linear-programming-duality/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/linear-programming-duality/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enEvery linear programming problem has a corresponding dual problem. The dual is itself a linear program, and the dual of the dual is the original problem. The optimal solution of the primal can be obtained from the dual; sometimes solving a linear program via duality theory is simpler and reveals the essence of the problem more clearly. Inspired by duality theory, the performance of the simplex method has been improved, and some non-simplex methods for solving linear programs have emerged, which this article does not cover in detail.Fri, 11 May 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/cpp-parameter-passing/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/cpp-parameter-passing/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enIn a C++ program, calling a function requires passing it arguments. Apart from an empty parameter list (void), parameter passing comes in two kinds: pass by reference and pass by value.Fri, 04 May 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/simplex-algorithm/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/simplex-algorithm/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enIn 1947, Dantzig proposed a method for solving linear programming problems, now known as the simplex method. It is a concise and efficient algorithm, hailed as one of the ten algorithms with the greatest impact on scientific development and engineering practice in the 20th century.Fri, 04 May 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/linear-programming-intro/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/linear-programming-intro/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enAmong optimization problems there is a class known as linear programming problems, which belong to constrained optimization. Linear programming is the problem of finding the extremum of a linear objective function under linear constraints (equalities or inequalities).Fri, 27 Apr 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/cpp-const/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/cpp-const/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enWhen programming we often need to define a kind of variable whose value never changes, for example pi=3.14, e=2.72, or the elastic modulus of a material. That is when the const keyword comes in.Thu, 26 Apr 2018 00:00:00 GMTTechnologyhttps://frankk.site/en/blog/genetic-algorithm-pid-tuning/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/genetic-algorithm-pid-tuning/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enAutomatically tuning PID parameters with a genetic algorithm in MATLAB / Simulink: building the simulation platform, designing an error-centric objective function, writing the interface function between the GA and Simulink, and iterating with a main function. Of limited practical use, but worth noting as a learning method.Wed, 29 Mar 2017 00:00:00 GMTGenetic AlgorithmsPIDSimulinkhttps://frankk.site/en/blog/genetic-algorithm-matlab/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/genetic-algorithm-matlab/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enWalking through the complete genetic-algorithm pipeline on a classic problem—finding the maximum of a two-variable function—covering individual encoding, the initial population, fitness, selection / crossover / mutation, with a MATLAB implementation and a quick reference for the most-used functions in the Sheffield GA Toolbox.Mon, 27 Mar 2017 00:00:00 GMTGenetic AlgorithmsMATLABOptimization Algorithmshttps://frankk.site/en/blog/matlab-vehicle-dynamics-curves/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enhttps://frankk.site/en/blog/matlab-vehicle-dynamics-curves/?utm_source=rss&utm_medium=rss&utm_campaign=rss_enA big problem from my Automotive Theory class: given the parameters of a medium-duty truck, use MATLAB to compute and plot the driving-force / road-resistance balance diagram, the acceleration-time curve, the dynamic-factor diagram, and the power-balance diagram—and along the way find the top speed and the maximum gradeability.Sun, 26 Mar 2017 00:00:00 GMTMATLABAutomotive TheoryVehicle Engineering