Optimizing Firefox: Why Large Applications Start Slowly on Linux

Optimizing Firefox: Why Large Applications Start Slowly on Linux

Taras Glek - taras@mozilla.com

https://blog.mozilla.com/tglek

Why Improve Startup Time?

Browser always starts fast for developers
Relatively rarely explored optimization
Turns out Firefox, Chrome, OpenOffice, etc suffer from the same problem

Measuring Startup Time:IO Hurts

When does application start?
Before main, before _main, before static constructors
Linux provides a tricky to get at timestamp in jiffies!
Warm 500ms+
Cold 3s-30s
CPUs get faster, a lot of slow storage devices remain

Visualizing Startup Time

.rel.dyn/.rela.dyn
.text
.rodata
blog post

Visualizing MMAP IO: Systemtap to the rescue

Systemtap allows us to hook into interesting kernel functions to observe a physical system
Everything else says how many major faults occured
IOGrind: IO simulator
Icegrind: high precision modelling of page faults to feed the linker

Who to blame?

Firefox? Is Mozilla incompetent?
Distributions? Is Firefox being packaged wrong
ld.so? Are libraries loaded wrong?
gcc/ld? Is the linker arranging the binary wrong?
Linux? Why are we loading 20MB of data in 128K chunks (while the cpu governor runs amok)?

Demand-paging is a strange way to load software

Disk IO pattern is a side-effect
Extrimely hard to connect code that faulted and resulting block IO
Every time a new page is faulted, VM requests readaround for 128K of data from disk

Demand-paging + CPU Scaling = OOF

CPU is effectivey idle while servicing hard faults
Low frequency + CPU-scaling overhead: 8% to 13% difference in startup time
Busy-waiting is a speedup

Tricky Solution: order the binary

	Startup Time	RSS (KB)
firefox.stock	2515ms	49452
firefox.ordered	1919ms	45344
firefox.static	2321ms	49616
firefox.static.ordered	1577ms	37072
firefox.static.pgo	1619ms	38436

Blog post

GCC + Linker Improvements

Paging backwards hurts
GCC 4.6 groups static constructors. With LTO they are also inlined
GCC 4.7 will run static constructors forward
Wanted: GCC PGO should record the order of invocation of functions and use --section-ordering-file to order resulting binary efficiently

Wanted: Linker Improvements

ELF workarounds with elfhack:

Platform	time spent before `XRE_main` without relocations packing (ms)	with relocations packing (ms)
GNU/Linux x86	1,362	1,273
GNU/Linux x86-64	1,643	1,318
Maemo 5, n900	1,717	1,427
Android 2.2, HTC Desire	4,250	3,568

Should be able to mark heavily used libraries for ld.so to fadvise()

Wanted: Linux Paging Improvements

Ability to page applications in larger chunks
Vectored madvise too?
Should put off relocations until a page is brought in from disk
While at it why not support demand-paging of compressed data?
Anything else that will increase disk throughput during startup

Fragmentation: Sparse files considered harmful

Linux makes it easier to fragment files than Windows!
XFS broke up files into 1000s of fragments (perhaps BTRFS is worse?)
SQLite is the worst offender (SQLITE_FCNTL_CHUNK_SIZE is your friend)

Windows: Sucks in Different Ways

Less libraries to load
No sparse files
Much better profiling tools for IO. See xperf
No significant relocation overhead?
Pages stuff in laughably small 32K(for executable) and 16K(for data) read-arounds
Slower IO

What's implemented?

Hard to deploy binary ordering without PGO
firefox dlopens libxul.so after fadvise(WILLNEED)
Elfhack reduces the size of our binary significantly
Custom ld.so on android

Summary

We know why applications start slowly
Please help us optimize Linux plumbing to start applications efficiently

Thank you