At 20, Linux is invisible, ubiquitous

[scandal 951]

Happy birthday, Linux.

My first Linux experience was with Yggdrasil, around 1996. About 5 years after Torvalds wrote the famous comp.os.minix post that started it all.

Today I run Mint at home and we run Suse on our servers where I work. We still run Solaris on our desktops though

At 20, Linux is invisible, ubiquitous

By Mark Milian, CNN

August 26, 2011 -- Updated 1658 GMT (0058 HKT) | Filed under: Gaming & Gadgets

San Francisco (CNN) -- Ross Turk would be happy to explain the tattoo on his arm.

By now, he's used to the penguin being met with bewildered stares. It represents, as he'd tell you, the Linux computing software, not the slightly less obscure character from 1950s cartoons.

"A lot of people see it and they think it's Chilly Willy or something," the West Hollywood, California, man lamented in a recent interview. "The Linux logo is still kind of grass-roots."

When the then-21-year-old Turk got the logo etched into his left bicep in 2000, the penguin seemed poised to become mainstream, then appearing frequently in magazines and on the walls in computer stores. But the software market tumbled with the dot-com bust, and so too did the Linux brand, choked by investors' swift rejection then of the open-source software movement.

Thursday marks 20 years since Linus Torvalds announced on a Web bulletin board that he'd begun working on a free computer operating system. In that message, Torvalds described Linux as "just a hobby, won't be big and professional."

Now, two decades later, that market breakthrough doesn't seem any more attainable. And yet while the Linux name and its penguin mascot failed to go big, the software they embody is more pervasive today than ever.

Linux's skeleton and spirit live on inside another familiar, adorable mascot: the green robot that represents Google's Android operating system. That software, which powers 43% of smartphones worldwide, many tablets and the Google TV set-top boxes, was developed with Linux at its core. Google's Chrome OS for laptops is also based on Linux.

Another mobile system, webOS, sprouted from Linux. Hewlett-Packard says webOS, not the hardware that runs it, is a key asset from its acquisition last year of Palm. This month, HP took steps to discontinue its gadget production arm, but it will keep webOS. HP has discussed licensing the software to other vendors in order to expand webOS's reach, perhaps into computing platforms on appliances and in cars.

Linux is already commonly installed on refrigerators with built-in TVs, car navigation systems, in-flight entertainment systems, public transit displays, ATMs and countless other machines. The Smart TV from Samsung Electronics, which competes with the Google TV, is also based on Linux. Sony previously allowed tinkerers to install versions of Linux onto their PlayStation consoles.

Whether you're aware of it or not, Linux is practically everywhere.

"The fact that you don't have to call it Linux is what makes Linux work," said Jim Zemlin, the executive director of the Linux Foundation. His nonprofit organization was formed to promote Linux development to the industry and sponsors Torvalds' ongoing work on the platform.

14 million lines of code

Linux can exist in so many places because, rather than being owned by one company, thousands of engineers contribute code to the kernel. (The kernel is the brains and sinew of the software, and Torvalds said in an e-mail that it's the aspect of his work that he finds most interesting and that he spends most of his time developing.)

No one can claim ownership of Linux, and everyone is free to use it. The software contains 14 million lines of code and is protected by more than 520,000 patents, according to a Linux Foundation report. Governments like the system's flexibility and decentralized nature.

Technology companies, even giants like Intel and AMD that typically don't publish schematics for their other products, encourage staff to contribute to and implement code from Linux. Google has carried this philosophy into many parts of its business, though not the ones that make the most money. The company did not respond to a request to make an executive available.

Torvalds initially conceived of Linux as a free alternative to Windows. But the collaborative-development, peace-loving ideologies of Linux were no match for the freewheeling, business-savvy, marketing power of Microsoft.

Linux, as a PC platform in the home, showed promise during the boom a decade ago. But it never came to fruition there, even as Apple's Mac has emerged as a more serious player.

Instead, Linux became the bastion of geek morality, the king of the fast-growing server industry where Microsoft and Apple also compete with limited success, and the choice platform for supercomputers in laboratories.

In Microsoft's annual report filed last week to the U.S. Securities and Exchange Commission, the software giant revised its statement on competition to declare the war with Linux over. Microsoft no longer perceives Linux as credible competition to Windows, the change suggests, as ZDNet notes.

A 'sticky' environment

However, Torvalds isn't ready to forfeit the PC.

"I'm definitely not indifferent to the desktop market," Torvalds wrote in an e-mail. "The desktop is a very 'sticky' environment: Users really get attached to their environment."

Several Linux players are still tackling that market, but their efforts amount to only about 1% of desktop usage worldwide. Microsoft controls the lion's share. Microsoft has been very adept at ensuring that Windows comes as the default operating system installed on most new computers.

"Usage isn't what matters; mindshare is what matters," said Jono Bacon, a community manager for Ubuntu, the most popular general-purpose version of Linux. "The biggest challenge we face right now is getting preinstalled on hardware."

You can't even fly on an airplane ... you can't use Facebook, you can't buy a book from Amazon, without running into Linux.

SUSE, which makes another Linux desktop platform, and others have been choked by Microsoft's "strong monopoly on the desktop," said Alan Clark, the chariman of the board for the openSUSE Project.

"It's played out differently than I expected, to be honest," Clark said. "We made some progress, but nothing like anybody envisioned."

Yet, SUSE has a comfortable presence in the server market, Clark said.

"Linux is very much pervasive. It's everywhere. You can't even fly on an airplane; ... you can't use Facebook; you can't buy a book from Amazon," Clark said, "without running into Linux."

The cult of Linux

Familiarity with Linux became a crucial skill for budding software engineers and server caretakers as far back as the mid-1990s.

When David Bohnett sought a partner in his new Web venture called GeoCities, resulting in one of the largest Internet business deals ever when it went public and then was acquired by Yahoo for $3.6 billion in 1999, his main criteria was an adeptness with Linux programming, Bohnett said in an interview. John Rezner fit the description and shared in the pair's eventual fortune.

Torvalds, the brain behind Linux, never seemed very interested in fortunes, according to people who know him. The reclusive programming wizard declined through a spokeswoman to be interviewed by phone, though he talked openly through e-mail and appeared on stage last week at LinuxCon in Vancouver, Canada.

There, Torvalds was treated like a celebrity. A lover of reclusive scientists, including Sir Isaac Newton and Albert Einstein, he described the general reactions from Linux fans as "just odd." He wrote: "Sometimes it does get to be a bit overwhelming."

Clark, from SUSE, described a memorable meeting with Torvalds.

"The first time he came to Japan, seriously, it was like a rock star arrived, and I could kind of tell it was really overwhelming for him," Clark said. "He took it in stride."

The Linux faithful are predominantly male, often nerdy, with strong principles about collaborative development that translate to a belief in a less hierarchal, more cooperative society, according to interviews. For example, Ubuntu's Bacon has an Android phone, which uses Google's open-source software, because "the ethical side of me feels like it's the right thing to do," he said. "It's not just a product. It's an ethos."

Apple has tried to define its principles in advertising: artistic, noncorporate and able to "think different." Its brand has been adopted by millions of people.

Meanwhile, Linux has maintained a devout but small following over the last two decades. The Linux software is embedded in many millions of machines, but its ethos and the penguin logo that embodies it remain an underground movement.

To let Turk explain it, because he'd be more than happy to, his tattoo is like a secret handshake, waiting for someone to be able to recognize and reciprocate.

"Every so often, at the gym or something, I'll run into someone who's like, 'That's the Linux tattoo,' and there's a conversation," said Turk, who now works for open-source software maker Talend. "It's always been something that the community feels like it owns. It's almost a little bit anti-establishment. The penguin doesn't stand a chance against the marketing of big firms. But that's great. It's good. It's ours."

[웃 910]

I have Ubuntu. My brother installed it years and years ago.

It's ok. The good is you cannot get any viruses.

The bad is it is difficult for me to install simple upgrades/installations/etc. On Windows everything is automatic.

And I really hate GIMP compared to Windows PAINT. Complete garbage and hard to use.

[웃 910]

Oh, the other thing about LINUX is that my clock is incredibly inaccurate. I literally have to update it every couple days

Right now I have 5:32 and my LINUX clock shows 5:28pm (time to update again). If I let it go for a few months without updating come end of November my LINUX clock would probably be showing early November or late October

It's annoying. Compared to the worlds most accurate clock which can run for the next 138 million years and only lose (or gain) less than a second.

[rsn 135]

I started using Slackware when I was in my teens.

I've given up on Ubuntu. Debian is an excellent server distro though. Ubuntu is a fork of Debian with a more aggressive release schedule.

Since I develop customer-facing software, I've found it helpful to work on the OS most customers use: Windows. I don't apply the same logic when it comes to browsers though.

Edited August 28, 2011 by rsn

[mawilson 2,133]

We still run Solaris on our desktops though

Solaris is a much superior OS, IMO.

ZFS, DTrace, Sun Studio (for development, though they do have it for Linux), stable ABI, clean, well-documented APIs.

No distro/kernel/glibc/whatever versioning nightmare that you get with Linux.

Oh, the other thing about LINUX is that my clock is incredibly inaccurate. I literally have to update it every couple days

You should install "ntpd".

[scandal 951]

Solaris is a much superior OS, IMO.

With the purchase by Oracle the future for Solaris looks ominous. On x86 systems modern Linux distros, particularly RH5 and Suse run circles around it. Also, the nifty acceleration technologies like kernel-bypass NICs, RDMA and Infiniband are either not supported on Solaris or are afterthoughts. Linux is where all that development is happening.

You should install "ntpd".

Yup. Clock drift is easy to solve.

[rsn 135]

With the purchase by Oracle the future for Solaris looks ominous.

oracle is not favored by the Java community either

[mawilson 2,133]

With the purchase by Oracle the future for Solaris looks ominous.

I'm not worried. Development is going full throttle and Solaris 11 is going to kick аss.

On x86 systems modern Linux distros, particularly RH5 and Suse run circles around it.

Not in my tests. All my Solaris boxes either outperform Linux or perform equally well, even with Solaris 10's aging TCP/IP stack. I expect Solaris 11 to be even better.

Also, the nifty acceleration technologies like kernel-bypass NICs, RDMA and Infiniband are either not supported on Solaris or are afterthoughts. Linux is where all that development is happening.

I have to agree, though the open source nature of Linux makes it easy to port a driver or a particular technology to Solaris, if need be.

[웃 910]

You should install "ntpd".

I looked that up and I show the latest version?

[scandal 951]

Not in my tests. All my Solaris boxes either outperform Linux or perform equally well, even with Solaris 10's aging TCP/IP stack. I expect Solaris 11 to be even better.

We've benchmarked our ticker plant and on all metrics Suse outperformed Solaris, on identical 12 core boxes. We did use Solarflare cards but without using kernel bypass, just straight NIC handoff to kernel and then feed handler buffers. Outbound NICs are also Solarflare, also without any hardware acceleration. It was a fair apples-to-apples test under real market conditions. We've done this on multicast full book feeds such as CME FIX/FAST and TV ITCH as well as TCP feeds such as Hotspot. CPU load was lower on Suse, average message latency was lower, and most importantly peak latency was lower. We simply see much less queuing on Suse. Needless to say we made the switch for the ticker plant, and have similar plans for the rest of the infrastructure. Desktops are further off but also in the plan. I envision a Solaris-free environment by some point next year.

[mawilson 2,133]

I looked that up and I show the latest version?

ntpdate is not ntpd.

ntpdate is something you run once to synchronize your clock with a time server.

ntpd is something that runs in the background all the time to keep your clock in sync. You should probably run ntpdate first because ntpd will not synchronize your clock if it's significantly off.

[mawilson 2,133]

We've benchmarked our ticker plant and on all metrics Suse outperformed Solaris, on identical 12 core boxes. We did use Solarflare cards but without using kernel bypass, just straight NIC handoff to kernel and then feed handler buffers. Outbound NICs are also Solarflare, also without any hardware acceleration. It was a fair apples-to-apples test under real market conditions.

Well... how many MSI-X interrupts (each bound to a core) per card? (I think Solar's default config has 2 per NIC.)

What was your intr_moderation value? Zero? rx_coalesce_mode?

Did you fan out your interrupts to multiple CPUs (set pcplusmp:apic_intr_policy=1 and set ip:ip_squeue_fanout=1)?

More importantly, how is your feed handler implemented on Solaris? A separate thread per multicast group + blocking recv(), a combination of /dev/poll + non blocking recv() or just non-blocking recv() in a loop?

Did you bind your process to a cpu set?

Did you change the scheduling class to real-time or FX (fixed priority)?

There are lots of tweaks you can do on Solaris to reduce latency.

[mawilson 2,133]

# echo ::interrupts -d | mdb -k

IRQ Vector IPL Bus Type CPU Share APIC/INT# Driver Name(s)

93 0x60 6 MSI-X 1 1 - sfxge#0

94 0x61 6 MSI-X 2 1 - sfxge#0

95 0x62 6 MSI-X 4 1 - sfxge#1

96 0x63 6 MSI-X 5 1 - sfxge#1

# intrstat 1

will tell you how often each core is interrupted.

[scandal 951]

Well... how many MSI-X interrupts (each bound to a core) per card? (I think Solar's default config has 2 per NIC.)

What was your intr_moderation value? Zero? rx_coalesce_mode?

Did you fan out your interrupts to multiple CPUs (set pcplusmp:apic_intr_policy=1 and set ip:ip_squeue_fanout=1)?

More importantly, how is your feed handler implemented on Solaris? A separate thread per multicast group + blocking recv(), a combination of /dev/poll + non blocking recv() or just non-blocking recv() in a loop?

Did you bind your process to a cpu set?

Did you change the scheduling class to real-time or FX (fixed priority)?

There are lots of tweaks you can do on Solaris to reduce latency.

I know some of the answers to those questions, for instance I know that we do bind our threads to cores and we do generally run dedicated thread per channel - at least where warranted. In some cases, that doesn't make sense. E.g. ICE charges for every price-server instance, with separate logins on the unicast request channel. So if you split each ICE multicast channel to a separate process you'd be paying for multiple logins. Given the traffic we see on ICE and our usage pattern that isn't warranted so we run multiple channels on single feed handler thread. For other markets we split more aggressively. I don't do the tweaking - we've got kernel folks who do the things you are mentioning and I'm comfortable that we've given Solaris every edge we could in the bakeoff. If anything, we tweaked Solaris more simply because we've been a Solaris shop for years and know it better - Linux is relatively new to us. We've been on Solaris since the 90s. There are people at our firm who are not without tears at the prospect of saying good bye. But we're seeing the writing on the wall for high performance, low latency computing. And Solaris is not in that future. Speak to your vendors - they'll likely tell you the same things. We are not the only shop making the switch, it's an industry wide move.